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Executive  Summary 


In  September  2008,  the  Boston  Mayor’s  Office  of  Emergency 
Preparedness  (MOEP)  tasked  us  with  identifying  a  suitable  performance 
measurement  system  for  their  organization.  Specifically,  the  policy 
question  they  asked  was:  What  is  the  best  way  the  City  of  Boston  can 
measure  the  performance  of  U.S.  Department  of  Homeland  Security- 
funded  initiatives  since  9/1 1?  Crafting  a  performance  measurement 
system  was  a  high  priority  for  MOEP,  but  they  lacked  the  necessary  staff 
and  resources  to  explore  a  solution  in-house.  In  response  to  MOEP’s 
request,  our  Policy  Analysis  Exercise  (PAE)  proposes  four 
recommendations  for  MOEP  as  they  move  forward  in  developing  a 
performance  management  system: 


I  1.  Implement  a  Balanced  Scorecard  approach 
I  2.  Focus  on  defining  and  executing  critical  priorities  using 
{  Richard  A.  Clarke’s  Metropolitan  Area  Security  Plan 

3.  Draw  from  DHS-style  measures,  but  also  focus  more  on 
creating  forward-looking  measures 

4.  Undergo  the  Emergency  Management  Accreditation  Program 

I  within  3  years 

i 


After  careful  review  of  current  performance  management  systems — 
particularly  those  measuring  preparedness  and  security — we  determined 
that  the  Balanced  Scorecard  is  the  most  practical,  applicable,  and 
sustainable  solution  for  MOEP.  Specifically,  we  believe  the  value  added 
of  the  Balanced  Scorecard  is  as  follows: 


The  Balanced  Scorecard  is: 

•  Simple 

•  Sustainable  and  has  long  shelf-life 

•  Proven  to  be  successful  in  public  and  private  sectors 

•  Immediately  ready  for  implementation 

•  A  holistic  approach  that  connects  all  activities  at  every 
organizational  level  with  overall  strategy 


To  ensure  that  we  chose  the  best  performance  measurement  system  for 
MOEP,  we  compared  the  Balanced  Scorecard  to  relevant  alternatives 
using  a  set  of  objective  criteria. 
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measure  intermediate  processes  and  might  indicate  milestones  in  the 
future,  in  addition  to  those  measures  that  reference  past  performance. 

Finally,  we  are  recommending  that  MOEP  ultimately  pursue  certification 
through  the  Emergency  Management  Accreditation  Program.  While  this 
program  will  be  costly  for  MOEP,  we  believe  it  will  provide  a  solid 
baseline  to  measure  MOEP’s  success  to  date,  and  will  provide  an 
additional  yardstick  of  accountability  for  MOEP  stakeholders. 
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SYSTEM 

AND 

CRITERIA 

Simple 

format 

Objective 

measurement 

Forward- 

looking 

measurement 

More  than 

mere 

checklists 

Inexpensive 

OMB  PART 
System 

In  addition  to  the  performance  measures  developed  within  Federal 
agencies,  the  President’s  Office  of  Management  and  Budget  (0MB) 
requires  performance  reporting  through  the  Program  Assessment  Rating 
Tool  (PART).  Developed  in  2004,  the  PART  program  is  a  uniform 
method  of  determining  performance  across  all  Federal  agencies,  helping 
0MB  to  measure  program  success  over  time  and  compare  the  results  of 
similar  programs.  As  0MB  describes  the  PART,  “[it]  looks  at  all  factors 
that  affect  and  reflect  program  performance  including  program  purpose 
and  design;  performance  measurement,  evaluations,  and  strategic 
planning;  program  management;  and  program  results.”'*’  Results  of  the 
PART  are  posted  on  ExpectMore.gov,  enhancing  transparency  and 
accountability  for  Federal  agencies.  In  addition,  they  help  inform  OMB 
funding  decisions  and  programmatic  modifications. 

To  assess  programs  using  the  PART,  OMB  asks  each  program  to  answer 
25  short-answer  questions  in  a  predominately  yes/no  format  (a  sample  of 
these  questions  are  shown  in  Figure  10  below).  Based  on  agency  answers, 
OMB  then  assigns  a  rating  to  each  program:  Effective,  Moderately 
Effective,  Adequate,  Ineffective,  or  Results  Not  Demonstrated.  In  FY 
2008,  OMB  ranked  16%  of  U.S.  Department  of  Homeland  Security 
programs  as  Effective,  34%  as  Moderately  Effective,  28%  as  Adequate, 
and  21%  as  Results  Not  Demonstrated  under  their  measurement  system. 
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Figure  10:  Sample  questions  from  OMR 's  PARl^^ 

•  Does  the  program  address  a  specific  and  existing  problem,  interest 
or  need? 

•  Is  the  program  designed  so  that  it  is  not  redundant  or  duplicative  of 
any  other  Federal,  State,  local  or  private  effort? 

•  Does  the  program  have  a  limited  number  of  specific  long-term 
performance  measures  that  focus  on  outcomes  and  meaningfully 
reflect  the  purpose  of  the  program? 

•  Does  the  program  have  ambitious  targets  and  timeframes  for  its 
long-term  measures? 

•  Does  the  program  (including  program  partners)  achieve  its  annual 
performance  goals? 

•  Are  independent  evaluations  of  sufficient  scope  and  quality 
conducted  on  a  regular  basis  or  as  needed  to  support  program 
improvements  and  evaluate  effectiveness  and  relevance  to  the 
problem,  interest,  or  need? 

•  Are  budget  requests  explicitly  tied  to  accomplishment  of  the  annual 
and  long-term  performance  goals,  and  are  the  resource  needs 
presented  in  a  complete  and  transparent  manner  in  the  program's 
budget? 

•  Does  the  program  use  strong  financial  management  practices? 

•  Has  the  program  demonstrated  adequate  progress  in  achieving  its 
long-term  performance  goals? 

•  Does  the  program  demonstrate  improved  efficiencies  or  cost 
effectiveness  in  achieving  program  goals  each  year? 

In  a  PART  assessment,  0MB  scores  each  of  four  areas:  Program  Purpose 
and  Design,  Strategic  Planning,  Program  Management,  and  Program 
Results/Accountability.  Here  we  highlight  the  results  of  one  program,  the 
Federal  Emergency  Management  Agency’s  Grants  and  Training  Office 
Assistance  to  Firefighters  Grants  program,  as  an  illustration.  In  2007,  this 
program  was  able  to  answer  “Yes”  to  most  PART  questions,  receiving  an 
“Effective”  rating,  and  scoring  high  marks  for  management  and 
performance  measurement. 


Figure  11:  Example  2007  PART  score 


Section 

Score 

Program  Purpose  &  Design 

80% 

Strategic  Planning 

88% 

Program  Management 

100% 

Program  Results/Accountability 

100% 
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Similar  to  the  Balanced  Scorecard,  OMB’s  PART  system  looks  at  more 
than  financial  indicators  to  assess  federal  programs.  Unlike  the  Balanced 
Scorecard,  however,  the  PART  program  does  not  set  specific  performance 
measures  -  rather,  it  simply  asks  if  agencies  are  meeting  their  targets. 

This  could  lead  to  a  situation  where  an  agency  is  successfully  meeting 
their  targets  each  year,  but  where  the  measures  are  not  linked  to  a  long¬ 
term  vision  or  strategy.  In  addition,  certain  PART  questions  such  as 
“Does  the  program  have  ambitious  targets  and  time  frames  for  its  long¬ 
term  measures?”  are  ambiguous  and  open  to  interpretation.  Without  more 
targeted  questions,  managers  could  easily  be  subjective,  picking  and 
choosing  which  areas  to  highlight  when  answering  the  questions.  Further, 
though  the  PART  provides  a  high-level  performance  overview  for 
program  managers  and  OMB  analysts,  it  does  little  in  terms  of 
communicating  the  organization’s  vision  and  strategy  to  lower-level 
employees,  the  public,  and  other  stakeholders.  Finally,  the  PART  system 
omits  a  “Learning  and  Growth”  perspective  to  measure  the  development 
of  agency  employees. 

National  Incident  Management  System  (NIMS) 

The  National  Incident  Management  System’s  (NIMS)  performance 
evaluation  system  fails  to  incorporate  objective  measurement  criteria, 
forward-looking  measures,  and  reach  beyond  mere  checklists. 


SYSTEM 

AND 

CRITERIA 

Simple 

format 

Objective 

measurement 

Forward- 

looking 

measurement 

More  than 

mere 

checklists 

■  ■ 

Inexpensive 

NIMS 

The  National  Incident  Management  System  provides  a  unified  approach  to 
emergency  situations  that  integrates  federal,  state,  tribal,  and  local 
governments,  as  well  as  the  private  and  not-for-profit  sectors  to  effectively 
carry  out  preparedness  and  response  activities.  NIMS  ensures  a  standard 
command  and  control  structure  and  facilitates  cooperation  between  entities 
at  different  levels  of  government  and  different  sectors  during  times  of 
crisis.  It  also  assesses  a  government’s  ability  to  reach  the  goals  of 
coordinated  preparation,  prevention,  mitigation  and  response  activities  by 
issuing  compliance  objectives  at  the  beginning  of  each  fiscal  year.  In 
FY08,  NIMS  issued  27  compliance  objectives,  24  of  which  had  been 
instituted  in  previous  years  but  are  required  to  be  continually  met  by 
governments  each  year.  Examples  of  objectives  for  FY08  for  local 
governments  are  listed  below. 
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Figure  12:  Examples  of  FY08  NIMS  Compliance  Objectives 


CATEGORY 

COMPLIANCE  OBJECTIVE 

Adoption 

Adopt  NIMS  for  all 
Departments/Agencies;  as  well  as 
promote  and  encourage  NIMS  adoption 
by  associations,  utilities, 
nongovernmental  organizations  (NGOs) 
and  private  sector  emergency 
management  and  incident  response 
organizations. 

Preparedness:  Planning 

Promote  and/or  develop  intrastate  and 
interagency  mutual  aid  agreements  and 
assistance  agreements  (to  include 
agreements  with  the  private  sector  and 
NGOs). 

Preparedness:  Training 

Use  existing  resources  such  as 
programs,  personnel  and  training 
facilities  to  coordinate  and  deliver 

NIMS  training  requirements. 

Preparedness:  Exercise 

Incorporate  NIMS  concepts  and 
principles  into  all  appropriate 
State/Territorial  training  and  exercises. 

Communications  and  Information 
Management 

Apply  common  and  consistent 
terminology  as  used  in  NIMS,  including 
the  establishment  of  plain  language 
(clear  text)  communications  standards. 

Resource  Management 

Inventory  response  assets  to  conform  to 
NIMS  National  Resource  Typing 
Definitions,  as  defined  by  FEMA 

Incident  Management  Systems 

Division. 

Until  FY  2007,  compliance  was  measured  by  self-certification  by  the 
governor’s  office  for  each  state  and  territory.  Beginning  in  FY  2007, 
NIMS  transitioned  to  performance-based  questions  and  a  measurement 
system  that  included  relevant  metrics  to  determine  whether  entities  met  the 
required  objectives.  In  2005,  NIMS  issued  the  NIMS  Capability 
Assessment  Tool  (NIMSCAST)  for  governments  at  each  level  to  assess 
and  report  on  their  compliance  with  the  stated  objectives.  The  questions 
that  determine  compliance  are  derived  directly  from  requirements  and  are 
asked  in  a  “Yes/No”  format  on  NIMSCAST.  For  example,  the  first 
objective  under  “Adoption”  is  measured  below. 
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Figure  13:  Example  NIMS  compliance  metrics  for  Objective  I :  Adoption^" 

Compliance  Objective  1:  Adopt  NIMS  for  all 
Departments/Agencies;  as  well  as  promote  and  encourage  NIMS 
adoption  by  associations,  utilities,  nongovernmental  organizations 
(NGOs)  and  private  sector  emergency  management  and  incident 
response  organizations. 

1.1  Has  the  local  government  formally  adopted,  for  FY08,  the 
National  Incident  Management  System  as  its  all-hazards 
incident  management  system? 

[  ]  Yes  [  ]  No 

What  legal  authority  was  used  to  formally  adopt  NIMS:  [Check 
all  that  apply.] 

[  ]  Executive  Order 

[  ]  Proclamation 

[  ]  Resolution 

[  ]  Other  legal  authority 

When  was  NIMS  formally  adopted?  [  ] 

1 .2  Does  formal  adoption  of  NIMS  expire? 

[  ]  Yes 

When?[  ] 

Is  there  a  process  for  renewing/maintaining  the  formal  adoption 
ofNIMS? 

[  ]  Yes  [  ]  No 
If  yes,  how?  [  ] 

1 .3  Does  the  local  government  promote  NIMS  adoption  by 
associations,  utilities,  private  sector,  and  nongovernmental 
organizations? 

[  ]  Yes  [  ]  No 


By  using  this  format,  NIMS  assesses  whether  or  not  governments  are 
complying  with  the  objectives  it  declares  at  the  beginning  of  each  fiscal 
year.  This  system  can  be  subjective  and  forces  organizations  to  meet  a 
checklist  of  requirements  that,  in  many  cases,  are  high-level  requirements 
that  lack  specificity.  It  may  be  difficult  to  come  to  an  overall 
determination  of  “Yes”  or  “No”  on  meeting  objectives  because  there  are 
many  disparate  parts  that  are  involved  in  each  objective  and  there  is  no 
mechanism  to  judge  whether  an  organization  is  performing  at  a  45%  level 
of  success,  versus  80%  or  99%. 
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U.S.  Department  of  Energy:  Los  Alamos  National  Laboratory 

The  Department  of  Energy’s  National  Laboratory  at  Los  Alamos,  New 
Mexico  has  done  extensive  work  in  developing  technologies  that  enable 
the  country  to  respond  to  terrorist  attacks  more  effectively.  We 
interviewed  public  officials  at  Los  Alamos  National  Laboratory  (LANL) 
to  determine  how  they  are  measuring  security,  and  to  better  understand  the 
concept  of  quantifying  risk  as  a  performance  measure.  While  the  risk 
factor  for  a  given  city  is  useful  in  directing  preparedness  actions  and 
providing  a  snapshot  of  security,  we  do  not  propose  that  it  be  used  in  a 
vacuum,  nor  as  the  solitary  performance  measure. 

Dr.  Wiley  Davidson,  the  Program  Director  of  the  Threat  Reduction 
Directorate  within  the  Homeland  Security  Programs  at  LANL,  spoke 
about  measuring  security  and  preparedness  in  terms  of  the  risk  equation, 
which  is  detailed  below. 

Figure  14:  Risk  equation^ ^ 


Risk  =  _Threat  x  _Vulner ability  x  _Conse queue es 


The  effect  on  the  overall  risk  is  multiplicative.  The  coefficient  of  threat  is 
a  measure  to  be  determined  by  local  intelligence.  After  the  threat  is 
quantified,  the  vulnerability  is  the  amount  of  people  that  are  exposed  to  a 
particular  threat,  and  the  consequences  can  be  quantified  in  either 
casualties  or  dollars.  In  addition  to  quantitative  methods  of  evaluating 
threats,  there  are  also  qualitative  methods  that  prove  useful  due  to  the 
inherent  difficulty  in  measuring  uncertain  events,  such  as  natural  disasters 
or  terrorist  attacks.  Risk  models  are  useful  mechanisms  to  assess  the 
threats  to  a  city,  but  the  element  of  uncertainty  that  is  inherent  in 
determining  risk  proves  to  be  a  formidable  challenge.  LANL  has  done 
extensive  work  developing  models  to  address  the  uncertainty  inherent  in 
the  risk  equation.  Dr.  Davidson  spoke  to  the  methods  available  to  assess 
the  threat  of  a  natural  disaster  or  a  terrorist  attack. 

There  are  probabilistic  approaches  and possibilistic 
approaches.  We  have  a  whole  methodology  we 've 
developed  here  [at  Los  Alamos  National  Laboratory]  that 
uses  possibilistic  approaches.  It  allows  you  to  take  some 
constructs  like  "likely”  or  "unlikely"  and  turn  those  into  a 
more  hard  core  probabilistic  treatment.  It ’s  called  an 
Evolved  Logic  Methodology...  You ’ve  got  a  mixture  of 
quantitative  and  qualitative  information,  so  you  need  to 
methodologies  that  allow  you  to  compare  quantitative  and 
qualitative  information  in  risk  equations.^ 
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MOEP  stated  that  the  organization  has  confidence  in  the  same 
methodology  detailed  by  Dr.  Davidson,  and  bases  many  of  its  decisions  on 
the  same  risk  framework.  Looking  at  the  risk  a  city  faces  can  be  a  useful 
indicator  of  preparedness  and  response  capabilities  when  used  in  concert 
with  other  metrics. 

Quantitative  measure  of  outputs  derived  from  the  public  health  community 

Another  approach  to  assess  security  was  first  projwsed  by  the  public 
health  community  to  evaluate  their  work  on  resjwnding  to  bio-terrorism 
and  other  threats. Two  major  objectives  drove  their  endeavors;  (1)  to 
measure  the  ability  of  the  public  health  community  to  respond  to  all 
events — not  just  bio-terrorism — by  measuring  its  preparedness  for  other 
threats  such  as  West  Nile  Virus  and  Severe  Acute  Respiratory  Syndrome 
(SARS),  and  (2)  to  measure  preparedness  by  evaluating  the  pieces  of  the 
overall  processes  that  are  involved  in  prevention  and  response.^^  This 
measurement  tool  helps  organizations  measure  their  ability  to  prevent, 
prepare  for,  mitigate  and  respond  to  terrorist  and  man-made  disasters  by 
assessing  the  relevant  outputs  associated  with  each  outcome.  Glen 
Woodbury,  a  faculty  member  and  associate  director  of  Executive 
Education  Programs  for  the  Naval  Postgraduate  School’s  Center  for 
Homeland  Defense  and  Security,  applies  this  measurement  process  to  the 
security  preparedness  realm. 

Woodbury’s  theoretical  example  is  presented  in  Figures  15  and  16,  which 
details  the  process  of  measuring  outputs  as  a  means  of  quantifying 
outcomes.  The  first  step  is  defining  the  desired  outcomes,  followed  by 
defining  the  measurable  outputs.  An  outcome  is  directly  related  to  the 
overall  mission  of  an  organization.  Outputs  are  initiatives  that  are  put  in 
place  to  achieve  an  organization’s  mission,  or  outcome.^’  After  outputs 
are  defined,  each  receives  a  score  between  one  and  five  from  the  list 
below.  This  example  is  measuring  an  organization’s  ability  to  evaluate 
potential  threats. 

Figure  15:  OutpiU  scoring  system 


Score  Definitions 

0  =  No  effort  or  system  underway  nor  recognition  of  the  need 

1  =  Recognition  of  the  need  but  no  effort  or  resources  to  accomplish  the 
output 

2  =  Initial  efforts  and  resources  underway  to  achieve  the  output 

3  =  Moderate  progress  towards  accomplishing  the  output 

4  =  Sustained  efforts  underway  and  output  near  to  fulfillment 

5  =  Output  achieved  and  resources  devoted  to  sustain  the  effort 
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Figidre  16:  Output  scoring  matrix 


Outputs 

Score 

1.  Adopt/develop  a  model  to  assess  threats 

2 

2.  Ensure  collaboration  of  traditional  and  non-traditional 
investigative  agencies 

2.5 

3.  Create  a  lead  organization  to  oversee  threat  identification 

4 

4.  Develop  strategies  to  overcome  barriers  to  the  sharing  of 
information  and  intelligence 

2.5 

TOTAL  (20  points  possible) 

11 

The  scores  of  each  output  are  compiled  to  indicate  how  close  an 
organization  is  to  achieving  a  stated  outcome.  Twenty  points  are  possible 
in  this  hypothetical  measurement,  and  the  organization  achieved  a  score  of 
eleven  in  the  example  above.  One  could  say  that  this  entity’s  “increased 
ability  to  evaluate  threats  as  they  are  identified”  is  not  yet  realized  but  is 
progressing.^*  In  general,  this  system  has  the  tendency  to  be  complex, 
arbitrary,  and  disconnected  from  overall  strategy. 

Measuring  preparedness  via  human  resources 

In  December  2008,  we  interviewed  Brian  D’Andrea,  a  U.S.  Navy 
Emergency  Preparedness  Liaison  Officer,  and  Jeff  Phillips,  the  former 
Director  of  Emergency  Operations  for  the  State  of  New  Mexico.*^ 
D’Andrea  has  extensive  experience  responding  to  emergencies  in  the 
United  States,  to  include  Hurricane  Katrina  in  2005.  He  emphasized  the 
importance  of  communication  and  coordination  between  entities  in 
responding  to  natural  disasters  and  terrorist  attacks.  He  highlighted  the 
necessity  of  a  measurement  mechanism  to  assess  the  degree  to  which 
communication  between  local,  state,  regional,  and  federal  levels  has 
changed  as  a  result  of  the  application  of  DHS  funds.  An  example 
measurement  would  be  the  number  of  interagency  meetings  held  per 
month.  Another  possibility  would  be  difference  in  the  costs  spent  on  the 
Joint  Terrorism  Task  Force  (JTTF)  before  9/1 1  and  the  funds  applied  to 
enhance  it  through  the  creation  of  a  fusion  center  after  the  terrorist 
attack.^*^  This  can  be  assessed  through  historical  interviews,  cost  records, 
plans  and  agreements  of  commitment  to  the  actual  fusion  of  agencies. 


MOEP  indicated  the  importance  of  interoperability  of  communication  in 
its  overall  strategy,  citing  specifically  the  transition  of  its  emergency 
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operations  plans  to  the  Emergency  Support  Functions  (ESF)  framework. 
This  format,  used  by  the  state,  ensures  more  fluid  communication  because 
both  will  use  the  same  framework  and  the  same  language. 

In  addition,  D’ Andrea  and  Phillips  suggested  measuring  the  degree  of 
turnover  in  emergency  preparedness  and  response  organizations  as  a 
means  of  assessing  continuity.  D’ Andrea  raised  the  example  of 
determining  who  participated  in  the  fusion  center  in  terms  of  the 
individual’s  experience,  qualifications,  time  in  service,  etc.  immediately 
after  9/11.  As  time  progressed  and  turnover  occurred,  the  question 
becomes  whether  that  position  was  filled  by  a  person  of  the  same  caliber, 
or  rather  a  lower-level  and  less-experienced  public  servant.  This 
determination  would  indicate  a  commitment  to  the  fusion  capability  of  the 
city.  These  types  of  measures  could  be  valuable  to  MOEP  as  they  devise 
the  metrics  they  want  to  use  to  assess  their  internal  processes.  Paying 
attention  to  human  resource  issues  as  well  as  communication  and 
coordination  are  important  components  to  an  organization  achieving  its 
mission. 


Current  Efforts  in  Cities  Similar  to  Boston 


The  City  of  San  Francisco 

The  City  of  San  Francisco  in  northern  California  is  very  similar  to  Boston 
in  terms  of  population  size,  multiple  surrounding  jurisdictions,  and 
vulnerabilities  inherent  in  living  close  to  maritime  water  sources.  A 
review  of  this  city’s  performance  measurement  system  was  encouraged  by 
our  PAE  advisor,  Richard  Clarke.  The  City  of  San  Francisco  Controller’s 
Office  maintains  a  database  that  tracks  performance  measures  for  the  city 
and  compiles  data  on  the  services  the  government  renders  to  the  public. 
The  report  issued  by  the  Controller’s  Office  includes  performance  results 
for  prior  fiscal  years,  targets  and  midyear  projections  for  the  current  fiscal 
year,  and  targets  for  the  next  budget  year.”'  The  goals  and  performance 
measures  for  the  Emergency  Communications  Department  are  listed 
below. 
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Figure  1 7:  San  Francisco 's  Emergency  Communications  Departmenf^ 

saSfiMT  X'J*; 

EMERGENCY  COMMUNICATIONS 

Goal  1  Staff  emergency  communication  center  with  fully-trained  personnel 


Measures 

1 .  Number  of  new  dispatchers  to  complete  training 

2.  Number  of  new  call  takers  to  complete  training 

3.  Percentage  of  fully  qualified  staff  maintaining  continuing  education 
requirements 

Groal  2 
Measures 

Respond  quickly  to  incoming  calls 

1 .  Total  number  of  calls  answered  in  the  communication  center 

2.  Total  number  of  non-emergency  calls  answered  in  the  comm,  center 

3.  Percentage  of  emergency  calls  answered  within  10  seconds 

Goal  3 
Measures 

Minimize  abandoned  calls 

1 .  Percentage  of  emergency  calls  abandoned  in  the  communication  center 

2.  Percentage  of  non-emergency  calls  abandoned  in  the  comm,  center 

EMERGENCY  SERVICES  COORDINATION 

Goal  1  Exercise  emergency  response  capabilities 


Measures 

1 .  Number  of  functional  exercises  conducted 

2.  Number  of  tabletop  exercises  conducted 

3.  Number  of  unique  participants  in  discussion  based  exercises 

4.  Number  of  unique  participants  in  functional  exercises 

Goal  2 
Measures 

Coordinate  interagency  planning 

1 .  Number  of  planning  task  force  meetings 

2.  Number  of  disaster  council  meetings 

3.  Number  of  training  courses 

4.  Assessment  of  training  program  quality  from  attendee’s  perspective 

5.  Percentage  of  tasks  added  and  completed  towards  the  Master  Improvement  Plan 

6.  Number  of  outstanding  tasks  of  the  Master  Improvement  Plan 

7.  Number  of  Department  Emergency  Operations  Plans  submitted 

8.  Percentage  of  Department  Emergency  Operations  Plans  reviewed 

9.  Current  balance  of  Grant  funding 

Goal  3 
Measures 

Promote  community  preparedness  for  emergencies 

1 .  Number  of  preparedness  presentations  made 

2.  Number  of  brochures  distributed 

3.  Number  of  days  of  public  education  advertising 

NON  PROGRAM 

Goal  1  All  City  employees  have  a  current  performance  appraisal 


Measures 

1 .  Number  of  employees  for  whom  performance  appraisals  were  scheduled 

2.  Number  of  employees  for  whom  scheduled  performance  appraisals  were 
completed 
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At  the  time  this  report  was  issued  (June  2007),  the  San  Francisco 
Controller’s  Office  report  includes  data  from  the  previous  3  years,  as  well 
as  the  targets  and  performance  at  6  months  into  the  current  fiscal  year,  A 
complete  measurement  for  the  first  goal  within  the  Emergency 
Communications  category  is  shown  below. 

Figure  18:  Goal  I  measuremenf^^ 


'  Goal  1  Staff  emergency  communication  center  with  fully-trained  , 

personnel 

Measure  1 .  Number  of  new  dispatehers  to  complete  training  I 


2003-2004 

2004-2005 

2005-2006 

2006-2007 

2006-2007 

2006-2007 

2007-2008 

Actual 

Actual 

Actual 

Target 

6  mo.  Actual 

Projected 

Target 

20 

10 

13 

22 

5 

13 

22 

if 


Measure  Definition:  Hiring  and  training  fidly  qualified  dispatchers.  Measure  relates 
I  directly  with  deportment's  goal  to  maintain  "Civilianization,  which  is  the  1 

?  recruitment  and  training  of  civilian  personnel  to  answer  and  dispatch  fire  and  HMS 
V  calls.  \ 

Data  Collection  Method:  ECD's  Training  Unit.  ^ 

FY06~07  6-month  Actual  and  Projection:  The  FY  06-07  projection  accounts  for  a  i| 

lower  number  of  academy  recruits.  f 

?  Data  Frequency  and  Reporting  Date:  Monthly 

I  FY07-08  Target:  ECD  will  conduct  training  for  40  dispatcher  recruits  during  hY07- 
08.  Expected  student  retention  of  55%  for  this  8238  classification. 

The  ability  to  measure  current  progress  against  past  performance  enables 
the  City  of  San  Francisco  to  determine  with  a  higher  degree  of  certainty 
where  the  government  stands  in  terms  of  meeting  its  goals.  The  detailed 
and  consistent  recording  of  data  allows  the  city  to  conduct  trend  analyses 
and  determine  current  performance  against  past  achievement. 

Furthermore,  instituting  a  6  month  measurement  allows  the  city  to  track 
interim  progress  before  the  year’s  end.  This  serves  to  indicate  whether  the 
government  is  on  track  or  needs  to  speed  up  efforts  to  reach  its  goals. 

The  specificity  of  San  Francisco’s  performance  measures,  the  inclusion  of 
past  data,  and  the  interim  6  month  measurements  against  targets  all 
contribute  to  the  measurement  system’s  ability  to  assess  how  well  the 
government  provides  services  to  citizens. 

Richard  Clarke  suggested  we  compare  the  capabilities  in  cities  that  are 
similar  to  Boston  in  terms  of  population,  vulnerabilities,  and  geographic 
location  in  order  to  help  MOEP  benchmark  its  own  current  capabilities. 

Comparing  select  measures  from  the  City  of  San  Francisco’s  performance 
measurement  system  to  Clarke’s  Metropolitan  Area  Security  Plan  is 
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illuminating  and  provides  an  assessment  of  how  a  city  similar  to  Boston  is 
performing  in  the  emergency  preparedness  arena.  For  example,  the  San 
Francisco  government  focuses  on  reliable  and  resilient  communications 
systems.  The  Telecom  and  Information  Services  Department  tracks  what 
percentage  of  the  time  communication  systems  are  up-and-running.  The 
Mayor’s  emergency  telephone  system  is  up  98%  of  the  time  as  recorded  in 
2007.  The  Port  Department  does  not  track  any  performance 
measurements  related  to  security  in  the  Office  of  the  Controller’s  report. 
Surveys  make  up  a  significant  amount  of  data  collection.  For  example,  the 
average  satisfaction  with  airport  security  and  average  wait  time  in  security 
checkpoints  were  key  indicators  in  the  airport  measurements.  In  line  with 
Clarke’s  security  plan,  San  Francisco  devotes  much  time  and  energy  to 
training  activities,  both  mock  exercises  and  job-specific  training  and 
certifications.  The  San  Francisco  Fire  Department  strives  to  educate  the 
public  about  how  to  respond  to  emergencies  by  tracking  the  number  of 
presentations  made  to  the  citizens  and  the  number  of  response  pamphlets 
disseminated.  Agencies  that  share  in  emergency  response  tasks  have 
begun  the  process  of  convening  to  develop  shared  performance  measures 
to  track  their  progress.^"*  Determining  how  other  cities  similar  to  Boston 
are  ensuring  preparedness  and  measuring  performance  is  useful  to  MOEP 
as  they  move  forward  in  developing  their  own  internal  performance 
measurement  system. 

The  City  of  Baltimore  and  the  State  of  Maryland 

The  City  of  Baltimore  is  well-known  for  developing  CitiStat,  a 
performance  measurement  program  credited  with  dramatically  improving 
city  services.  In  fact,  just  four  years  after  its  inception,  Baltimore’s 
CitiStat  received  the  prestigious  Innovations  in  American  Government 
Award  from  Harvard  University  and  the  Council  for  Excellence  in 
Govemment.^^  The  CitiStat  program,  “regularly  reviews  every  aspect  of 
Baltimore  City’s  government  -  ranging  from  employees’  use  of  overtime 
and  sick  leave,  to  trash  collection  and  snow  removal  -  ensuring  that  the 
city  is  meeting  the  needs  of  its  citizens.  Bimonthly  meetings  between 
agencies,  the  mayor,  the  first  deputy  mayor  and  members  of  his  cabinet 
serve  as  a  key  component  of  CitiStat.  Before  the  meetings,  agencies  are 
required  to  submit  performance  reports,  which  are  compared  to  data  from 
the  city’s  311  Call  center,  an  independent  body  that  manages  service 
requests  from  citizens.”^  CitiStat  is  based  on  four  core  tenets. 

1 .  Accurate  and  timely  intelligence  shared  by  all 

2.  Rapid  deployment  of  resources 

3.  Effective  tactics  and  strategies 

4.  Relentless  follow-up  and  assessment 
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When  it  came  to  tracking  homeland  security  preparedness,  however. 
Mayor  Martin  O’Malley  (1999-2007)  chose  a  different  track.  CitiStat’s 
formulaic  and  service-oriented  approach  was  not  conducive  to  measuring 
homeland  security  preparedness,  which  required  more  long-term  strategy 
and  capability  planning.  In  an  interview  for  this  PAE,  Andrew  Lauland, 
former  Homeland  Security  Advisor  to  Mayor  O’Malley  and  current 
Homeland  Security  Advisor  to  now-Govemor  O’Malley,  described  the 
rationale  behind  their  approach,  which  was  later  continued  at  the  state 
level. 


As  much  time  as  we  spent  doing  CitiStat,  we  did  not  really 
stat  homelcuid  security  per  se.  For  homeland  security,  we 
had  a  playbook  of  basic  operational  capabilities  that  we 
thought  we  needed  to  build  up.^ 

This  model  is  very  similar  to  Clarke’s  Metropolitan  Area  Security  Plan, 
which  identifies  critical  preparedness  priorities  and  the  order  in  which 
cities  should  complete  them.  At  the  state  level,  Maryland  officials 
identified  twelve  critical  preparedness  goals,  and  are  currently  in  the 
process  of  developing  measures  and  targets  for  each  of  those  goals.  A 
complete  list  of  Maryland’s  twelve  core  homeland  security  goals  can  be 
found  in  Maryland's  Strategic  Goals  and  Objectives  for  Homeland 
Security. Two  of  the  goals  are  displayed  here. 

Figure  19:  I^xample  of  Maryland's  Core  Goals  for  Homeland  Security™ 


•  Interoperable  Communications — First  responders  in  every  region  in 
Maryland  should  have  access  to  a  fully  digital,  trunked  radio  system 
which  all  response  partners  can  access  in  order  to  transmit  and  receive 
voice  and  data.  First  responders  in  every  region  should  have  robust 
CAD/RMS  systems  capable  of  coordinating  dispatch  data  for  all 
response  partners  and  capable  of  transmitting  data  to  systems  such  as 
WebEOC  for  consolidation  and  roll  up  of  regional  CAD  data 


•  HAZ  MAT/Explosive  Device  Response — Every  metropolitan  region  5 
should  have  a  Type  1  Haz  Mat  team  and  a  Type  1  bomb  response  I 
team,  either  as  one  unit,  or  separate  units,  and  there  should  be  l| 
sufficient  units  statewide  to  provide  a  mutual  aid  response  in  any  ^ 
jurisdiction  within  a  minimal  amount  of  time.  These  teams  should  all 
be  trained  for  both  fire  and  law  enforcement  response. 


A  list  of  the  performance  targets  associated  with  the  second  goal  (HAZ 
MAT/Explosive  Device  Response)  is  provided  in  the  figure  below. 
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Figure  20:  Targets  within  Maryland’s  Core  Goal:  HAZ MA  T/Explosive 
Device  Response^‘ 


■  '5— I 


■  r' -  c,-\ r  .  nr .r;  ;  / : 


Every  metropolitan  region  in  Maryland  should  have  access  to  a  Type  1 
bomb  response  and  HazMat  team,  and  those  units  should  be  able  to 
provide  mutual  aid  within  a  minimal  amount  of  time. 


■S 


^  •  Define  a  standard  set  of  equipment  and  training  for  Bomb  Squad  and 

j  HazMat  Teams  in  Maryland  and  apply  that  standard  across  the  state, 

s  These  teams  should  meet  Type  1  standards,  but  Maryland’s  standards 

I  should  incorporate  or  consider  emergent  technologies  that  provide 

I':  enhanced  response  and  render  safe  capabilities. 

I  •  Establish  a  formalized  system  of  mutual  aid  to  ensure  every 
|;  jurisdiction  has  access  to  a  Type  1  bomb  response  and  HazMat  team 

I  within  a  determined  response  window. 

I  •  Key  data  including  response  times  and  incident  types  should  be  shared 
between  state  and  local  teams,  and  analyzed  for  more  effective 
f  response  on  a  regular  basis. 

|: 

I  State  and  Local  HazMat  and  Bomb  Teams  should  be  trained  for  both  fire 
and  law  enforcement  response  and  to  handle  unique  WMD/CBRNE 
I  incidents. 


I 


'S 


I  •  Identify  cross  training  needs  for  HazMat  teams  so  that  teams  can 
I  support  and  effectively  coordinate  with  all  law  enforcement  response 

*  (e.g.,  crime  scene,  evidence  collection,  safety). 

I  •  Develop  a  cross  training  plan  for  Bomb  response  teams  in  | 

I  WMD/CBRNE  response.  i 

?  '■  -  t 


Like  our  review  of  City  of  San  Francisco,  we  wanted  to  compare 
Baltimore’s  preparedness  goals  to  the  actual  capabilities  they  have 
achieved  thus  far.  Lauland  identified  several  priorities  that  have  been 
largely  met  in  Baltimore,  including  CCTV  covering  critical  infrastructure, 
covert  air  monitors,  specialized  drills  and  exercises  (such  as  Red  Envelope 
Drills,  where  police  are  given  surprise  practice  drills  in  red  envelopes), 
and  personal  protective  equipment  such  as  WMD  kits  in  the  trunk  of  every 
police  car.  Lauland  also  indicated  that  Maryland’s  priorities  adapt  with 
new  intelligence  and  information.  For  example,  recent  reports  indicated 
that  chlorine  has  been  used  in  Improvised  Explosive  Devices  (LEDs)  in 
Iraq.  In  response,  Maryland  officials  are  moving  to  secure  all  potential 
chlorine  deposits  in  the  state.’^ 
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Finally,  Lauland  noted  that  the  key  to  CitiStat’s  success  was  not  its 
innovative  software,  but  rather  the  intense  buy-in  and  engagement  by 
high-level  city  officials. 

It  matters  less  what  system  you  use  -  it 's  having  a  system. 

That  is  the  real  challenge.  There  was  nothing  magic  about 
CitiStat  -  the  magic  was  the  engagement  and  the 
accountability.  The  Mayor  went  to  meetings  every  week. 

You  could  have  picked  lauother  .system]  up,  if  you  .stuck  to 
it  and  had  executive  engagement.  CitiStat  wm’  never  about 
the  electronics. 

A  further  discussion  of  high-level  agency  engagement  with  respect  to  the 
Balanced  Scorecard  can  be  found  in  the  ensuing  Implementation  Plan. 


Implementation  Plan:  The  Balanced  Scorecard  and  MOEP 


If  the  Mayor’s  Office  of  Emergency  Preparedness  implements  the 
Balanced  Scorecard  provided  in  this  PAE,  certain  components  are 
important  to  the  implementation  process  that  will  ensure  a  successful 
execution  of  the  performance  measurement  system.  These  important 
points  include:  cascading  the  scorecard  to  create  alignment  at  every  level 
of  the  organization,  sharing  the  scorecard  with  stakeholders  with  the 
adequate  security  clearance  to  ensure  accountability  to  the  scorecard,  and 
adhering  to  implementation  principles  that  are  important  in  integrating  the 
scorecard  into  the  organization’s  measurement  processes. 

Developing  cascaded  scorecards  at  each  level  of  the  organization  to 
ensure  alignment  with  overall  mission 

Cascading  the  Balanced  Scorecard  to  lower  levels  within  an  organization 
refers  to  devising  scorecards  at  each  level  of  the  organization  in  an  aligned 
fashion  that  ensures  all  processes  within  the  agency  support  the  overall 
mission.  In  the  public  sector,  cascading  is  a  necessity  rather  than  an 
alternative,  as  many  different  entities  and  people  are  involved  in  achieving 
results.’"*  The  goal  of  cascading  is  to  provide  all  groups  within  the 
organization  the  opportunity  to  demonstrate  how  their  specific  actions 
contribute  to  overall  success.  The  cascading  process  begins  with  the 
overall  organizational-wide  scorecard,  and  then  additional  scorecards  are 
developed  at  increasingly  lower  levels.  For  example,  additional 
scorecards  may  be  devised  at  the  department-level,  the  team  level,  and 
finally  the  individual  level.  Figure  21  (below)  depicts  this  process. 
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Figure  21:  Cascaded  Balanced  Scorecard  Modef^ 


. .  - : 

Mission,  Values,  Vision,  Strategy 


Each  cascaded  scorecard  links  back  to  the  organizational-wide  scorecard; 
this  process  ensures  that  employees  at  the  every  level  are  working  toward 
achieving  the  overall  mission.  It  is  important  to  verify  that  alignment  is 
achieved  between  scorecards  at  each  level  and  that  the  measures  and 
objectives  at  lower  levels  do,  in  fact,  contribute  toward  the  overarching 
strategy. 

The  Mayor’s  Office  of  Emergency  Preparedness  should  devise  Balanced 
Scorecards  at  all  levels  of  its  organization,  possibly  around  functional  or 
jurisdictional  areas,  to  ensure  all  work  contributes  to  MOEP’s  overall 
vision  of  a  Metro-Boston  Homeland  Security  region  with  strengthened 
capabilities  to  ensure  its  safety  and  security.  This  can  be  achieved  by 
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convening  jurisdictional  leaders  together  so  that  MOEP  leadership  can 
lead  a  strategy  planning  session  similar  to  the  one  we  conducted  with 
Donald  McGough  and  Atyia  Martin.  This  iterated  procedure  will  identify 
relevant  objectives,  measures,  targets  and  initiatives  to  align  processes 
more  closely  with  MOEP’s  overall  vision  at  each  level  of  the  organization. 
The  interview  questions  used  with  MOEP  leadership  can  be  found  in 
Appendix  B.  In  addition,  there  are  many  software  programs  that  assist  in 
generating  cascaded  balanced  scorecards  and  tracking  the  organization’s 
performance  against  the  measures,  which  are  available  through  an  online 
search.  Only  by  cascading  the  Balanced  Scorecard  throughout  the 
organization  will  MOEP  be  able  to  truly  streamline  and  align  all  processes 
at  each  level  within  its  organization  with  its  mission. 

An  example  of  a  cascaded  scorecard  for  the  Boston  Emergency  Operations 
Center  can  be  found  on  the  next  page. 
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Cascaded  Balanced  Scorecard  Example  for  Boston ’s  Emergency  Operations  Center  (EOC) 


Ensuring  the  organization  is  accountable  to  the  goals  laid  out  in  the 
Balanced  Scorecard 

Accountability  measures  are  an  essential  component  to  the  successful 
implementation  of  the  Balanced  Scorecard  in  an  organization.  Sharing  an 
entity’s  Balanced  Scorecard  with  stakeholders  on  a  quarterly  basis  is  a 
powerful  way  to  ensure  the  agency  is,  in  fact,  making  progress  toward 
achieving  its  objectives.  David  Bibo,  a  prior  employee  of  MOEP, 
addressed  the  issue  of  accountability  in  an  interview  for  this  PAE. 

Ihe  best  way  for  BSC  to  become  a  relevant  tool  for  MOEP 
is  for  them  to  feel  like  someone  else  is  holding  them 
accountable  to  it.  I'd  recommend  stiggesting  to  them  that, 
once  it 's  implemented,  they  share  the  scorecard  on  a 
quarterly  basis  with  the  regional  stakeholders  so  everyone 
knows  where  they  stand.  Unless  it's  shared  amongst  the 
customers  and  stakeholders,  it  will  "sit  on  a  shelf  and  not 
get  the  attention  it  deserves. 

It  is  highly  recommended  that  MOEP  ensure  accountability  by  sharing  its 
scorecard  and  performance  with  stakeholders.  It  is  not  expected  nor 
advised  that  MOEP  release  the  sensitive  information  that  is  included  in  the 
targets  and  initiatives  columns  of  the  scorecard  with  stakeholders  that  do 
not  have  adequate  security  credentials,  but  rather  that  the  organization 
share  its  progress  with  a  group  of  stakeholders  that  do  have  adequate 
security  clearance. 

Key  implementation  principles 

The  following  implementation  principles  provide  helpful  guidelines  when 
an  organization  puts  the  balanced  scorecard  into  practice:  (1)  cascaded 
scorecards  should  be  content-specific  to  represent  the  unique 
characteristics  of  each  group,  (2)  terminology  must  be  consistent 
throughout  organization,  (3)  limit  the  number  of  performance  measures  to 
only  the  critical  few,  (4)  personal  performance  objectives  should  be  linked 
to  the  Balanced  Scorecard,  (5)  include  Balanced  Scorecard  understanding 
in  key  leadership  competencies,  (6)  first-level  managers  are  vital  to  the 
acceptance  of  the  scorecard,  (7)  share  learning  as  a  team,  a  possible 
distribution  method  being  the  organization’s  Intranet,^’  and  (8) 
systematize  feedback  because  it  is  an  essential  component  to  generating 
momentum  within  an  organization.^*  Based  on  the  conversation  with  Utah 
State  government  officials,  we  recommend  that  MOEP  consider  allowing 
employees  to  develop  their  own  performance  measures.  In  their 
experience,  this  drastically  increased  employee  buy-in  and  encouraged 
employees  to  set  higher  goals  and  ardently  work  to  achieve  them.  MOEP 
might  also  follow  Utah’s  example  of  changing  the  traditional  Balanced 
Scorecard  perspectives  (Customer,  Financial,  etc.)  to  functional  areas 
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(Prepare,  Prevent  and  Respond).  Furthermore,  MOEP  should  strongly 
consider  linking  scorecard  objectives  to  its  budget.  Based  on  Baltimore’s 
experience,  high-level  buy-in  is  also  critical  to  a  successful  performance 
measurement  system. 

How  the  Balanced  Scorecard  is  implemented  in  an  organization,  which 
includes  incorporating  effective  accountability  measures,  will  facilitate  the 
success  of  the  performance  measurement  system  in  aligning  all  processes 
to  an  entity’s  overall  mission  and  assessing  progress  toward  achieving  that 
mission. 


Conclusion 


After  conducting  this  review,  we  recommend  the  following  to  MOEP  as 
they  move  forward  in  developing  a  sustainable  performance  measurement 
system  for  their  organization. 


1.  Implement  a  Balanced  Scorecard  approach 

2.  Focus  on  defining  and  executing  critical  priorities  using 
Richard  A.  Clarke’s  Metropolitan  Area  Security  Plan 

3.  Draw  from  DHS-style  measures,  but  also  focus  more  on 
creating  forward-looking  measures 

4.  Undergo  the  Emergency  Management  Accreditation  Program 
within  3  years 


By  building  and  deploying  a  Balanced  Scorecard  measurement  system, 
MOEP  will  be  able  to  track  its  progress  on  key  issues,  ensure  that  its 
budget  is  linked  to  its  overall  strategy,  and  meet  critical  needs  first. 
Through  a  robust  Balanced  Scorecard  system  with  cascaded  scorecards  for 
lower-level  parts  of  the  agency,  MOEP  can  become  a  cutting-edge 
performance  management  leader  within  the  City  of  Boston,  and  perhaps 
spur  other  city  agencies  to  follow  their  lead. 
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Appendix  A. 


We  interviewed  the  following  individuals  for  this  PAE: 

•  Donald  McGough,  Director,  MOEP 

•  Atyia  Martin,  Regional  Planner,  MOEP 

•  Professor  Richard  Clarke,  Harvard  Kennedy  School  of  Government 

•  Professor  Robert  Behn,  Harvard  Kennedy  School  of  Government 

•  Professor  Arnold  Howitt,  Harvard  Kennedy  School  of  Government 

•  Heather  Wilson,  U.S.  Representative,  State  of  New  Mexico 

•  Dr.  J.  Wiley  Davidson,  Program  Director,  Homeland  Security  Programs,  Threat 
Reduction  Directorate  at  Los  Alamos  National  Laboratory 

•  Dr.  Babs  Marrone,  Project  Leader  and  Principal  Investigator  in  the  Bioscience  Division  at 
Los  Alamos  National  Laboratory 

•  Chris  Ortega,  Los  Alamos  County  Utilities  Manager 

•  Steve  Yarbro,  Group  Leader  in  IAT-1,  International  and  Applied  Technology  Division 

•  JefTPhillips,  former  director  of  New  Mexico  Emergency  Operations 

•  Jay  Hone,  U.S.  Air  Force  Emergency  Preparedness  Liaison  Officer  (EPLO) 

•  Brian  D’ Andrea,  U.S.  Navy  EPLO 

•  Catherine  Holdeman,  Assistant  City  Manager,  Wichita,  Kansas 

•  David  Bibo,  U.S.  Secret  Service  and  former  Boston  MOEP  employee  working  homeland 
security  issues  under 

•  Mike  Hansen,  Director  of  Strategy  and  Management  for  the  Utah  Governor’s  Office 

•  Amy  Lightfoot,  Special  Programs  Director  at  the  Utah  Department  of  Public  Safety 

•  R.P.  Eddy,  CEO  Ergo  and  Senior  Fellow  for  Counterterrorism  at  the  Manhattan  Institute 

•  Andrew  A.  Lauland,  Governor’s  Homeland  Security  Advisor,  State  of  Maryland 
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Balanced  Scorecard  Interview  Questionnaire^^ 


Purpose  of  the  Organization 


1 .  What  is  the  mission  of  your  organization?  MOEP's  Mission  is  to  enhance  the  City  of 
Boston  and  Metro-Boston  Homeland  Security  Region's  capacity  to  prevent,  protect 
against,  respond  to,  and  recover  from  major  emergencies. 

2.  What  is  the  vision  of  your  organization?:  A  vision  statement  provides  a  word  picture  of 
what  the  organization  intents  ultimately  to  become  -  which  may  be  5,  10,  or  15  years  in 
the  future. 

3.  What  are  the  values  of  your  organization? 


Strategy 

“Your  strategy  is  the  common  thread  that  weaves  through  the  Scorecard  tying  the  disparate 
elements  of  customers,  processes,  employees,  and  financial  stakeholders  into  one  coherent 
whole.”  To  craft  a  strategy,  we  begin  with  a  SWOT  analysis,  which  looks  at  strengths, 
weaknesses,  opportunities,  and  threats  (graphic  pg.  173).  Utilizing  this  information,  we  will  craft 
a  one-page  “Strategy  Map”  that  graphically  depicts  your  organization’s  performance  objectives. 
A  Balanced  Scorecard  complements  the  strategy  map  by  once  again  outlining  the  objectives,  but 
also  providing  measures,  targets,  and  initiatives  used  to  gauge  success  on  the  objectives. 

1 .  What  are  MOEP’s  strengths? 

2.  What  are  MOEP’s  weaknesses? 

3.  What  opportunities  are  present  for  your  organization,  the  pursuit  of  which  will  lead  you 
to  your  mission? 

4.  What  threats  do  you  face  that  may  endanger  the  pursuit  of  your  mission? 

5.  Analysis:  Thinking  about  the  responses  to  the  four  questions  above,  what  are  your 
organization’s  strategic  issues?  Strategic  issues  are  defined  as,  “fundamental  policy 
questions  or  critical  challenges  that  affect  an  organization’s  mandate,  mission  and  values; 
product  or  service  level  and  mix,  clients,  users  or  payers;  or  cost,  financing,  organization, 
or  management.” 

6.  Given  these  strategic  issues,  what  are  your  strategic  priorities  for  each  of  these 
perspectives:  Customer,  Financial,  Internal  Process,  Employee  Learning  and  Growth? 


Balanced  Scorecard 
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NOTE:  Performance  measures  should  include  a  mixture  of  input,  output,  and  outcome  variables. 
In  addition,  there  should  be  a  mixture  of  lag  indicators  (things  that  have  already  happened;  and 
lead  indicators,  which  might  signal  milestones  in  the  future  -  these  measure  intermediate 
processes  and  activities). 

NOTE  II;  The  best  performance  measures  provide  direction,  align  employees,  improve  decision¬ 
making,  and  serve  as  a  basis  for  resource  allocation  decisions. 

Customer  Perspective 

1 .  Who  are  your  customers  or  stakeholders?  [We  have  identified  three:  Boston-area 
citizens,  the  Mayor’s  Office  of  Boston,  and  the  U.S.  Department  of  Homeland  Security] 

2.  What  are  your  objectives  for  each  segment  of  your  customers?  [Up  to  2  for  each 
segment] 

3.  How  might  you  measure  if  you  are  achieving  your  objectives? 

4.  What  is  the  ideal  target  for  each  of  those  measures? 

5.  Are  you  currently  undertaking  any  initiatives  to  meet  these  targets?  If  so,  could  they  be 
enhanced?  If  not,  are  there  initiatives  that  would  help  you  reach  these  targets? 


Internal  Perspective  [i.e.  innovation/operations/efficiency] 

1 .  To  continue  adding  value  for  your  customers  and  clients,  what  processes  must  you  excel 
at?  [Up  to  2] 

2.  After  analyzing  current  trends,  which  processes  might  you  be  expected  to  develop  and 
excel  at  in  the  foreseeable  future?  [Up  to  2] 

3.  How  might  you  measure  if  you  are  achieving  these  goals?  [possibilities  include  quality, 
innovation,  partnering]  NOTE:  Measures  in  the  Internal  Process  perspective  should  flow 
directly  from  your  choices  in  the  customer  perspective 

4.  What  is  the  ideal  target  for  each  of  those  measures? 

5.  Are  you  currently  undertaking  any  initiatives  to  meet  these  targets?  If  so,  could  they  be 
enhanced?  If  not,  are  there  initiatives  that  would  help  you  reach  these  targets? 


Learning  and  Growth  Perspective 

1 .  Which  skills  and  competencies  will  be  required  in  the  years  ahead?  [Up  to  2] 

2.  Do  your  employees  have  access  to  the  training  they  need  to  help  you  achieve  your 
customer  outcomes? 

3 .  Is  your  organizational  climate  conducive  to  success?  Do  you  have  a  strong  culture  and 
alignment  of  goals  throughout? 

4.  How  might  you  measure  if  you  are  achieving  these  goals?  [examples  include:  training  by 
level,  retention,  personal  development  plan  creation,  employee  satisfaction,  alignment] 

5.  What  is  the  ideal  target  for  each  of  those  measures? 

6.  Are  you  currently  undertaking  any  initiatives  to  meet  these  targets?  If  so,  could  they  be 
enhanced?  If  not,  are  there  initiatives  that  would  help  you  reach  these  targets? 
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Financial  Perspective 

1 .  What  financial  goals  need  to  be  achieved  to  realize  your  strategic  themes  and  objectives? 
[Up  to  2] 

2.  How  might  you  measure  if  you  are  achieving  these  goals?  [possibilities  include;  staying 
within  budget,  lowering  expenses,  increasing  return  on  investment,  increasing  cash  flow] 

3.  What  is  the  ideal  target  for  each  of  those  measures? 

4.  Are  you  currently  undertaking  any  initiatives  to  meet  these  targets?  If  so,  could  they  be 
enhanced?  If  not,  are  there  initiatives  that  would  help  you  reach  these  targets? 
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natural  or  man-made.”*  The  Director  of  MOEP  also  stated  the  vision  of 
the  organization  in  the  context  of  the  next  5-10  years  as:  “A  Metro-Boston 
homeland  security  region  with  strengthened  capabilities  to  ensure  its 
safety  and  security.”  The  values  of  MOEP,  also  stated  by  the  Director, 
include  transparency,  utilization  of  best  practices,  sustained  competencies 
of  preparedness  capabilities,  regional  collaboration,  all-hazards  based 
preparedness,  and  best  practices  adoption.  ^  MOEP  is  responsible  for  both 
the  City  of  Boston  and  the  Metro-Boston  Homeland  Security  Region 
(MBHSR),  managing  grant  funding  and  strengthening  preparedness 
programs.  Boston  receives  roughly  95%  of  its  budget  from  the  U.S. 
Department  of  Homeland  Security  in  order  to  improve  the  city’s  ability  to 
prepare  for  and  respond  to  natural  disasters  and  terrorist  threats.^  MOEP’s 
aim  for  this  PAE  is  to  acquire  performance  measurement  methods  and 
tools  to  assess  the  effectiveness  of  DHS-ftmded  security  initiatives  and 
their  impact  on  Boston’s  emergency  preparedness  and  response 
capabilities. 

The  need  for  performance  measurement  in  government 

A  central  responsibility  of  government  is  providing  services  to  citizens.  It 
is  essential  that  governments  at  every  level  be  able  to  measure  their 
performance  and  justify  their  use  of  taxpayer  dollars.  After  performance 
measures  made  waves  in  the  private  sector,  experts  began  to  see  utility  in 
the  public  sector.  In  his  article,  “The  Performance-Target  Ethic”,  Harvard 
Kennedy  School  Professor  Robert  Behn  addresses  how  citizens  can 
determine  whether  their  governments  are  performing  well.'*  Behn  asserts 
that  governments  often  attempt  to  communicate  progress  to  citizens  by 
posting  statistics  on  websites,  but  often  this  amounts  to  little  more  than  a 
plethora  of  inaccessible  and  non-essential  information  that  fails  to  give  the 
citizen  a  clear  understanding  of  government  performance.  Behn  advocates 
a  different  approach.  To  evaluate  the  performance  of  any  government, 
Behn  argues  that  citizens  need  to  answer  the  compared-with-what 
question:  “The  evaluation  of  a  public  agency  involves  a  comparison.  If 
you  decide  that  an  agency  is  outstanding,  you  do  so  because  it  is  better 
than  your  basis  of  comparison.  Thus,  every  evaluation  begins  with  the  key 
question:  ‘Compared  with  what?”’^  A  government  can  address  the 
compared-with-what  question  by  establishing  performance  targets — 
specific  goals  to  be  accomplished  by  an  organization  by  a  specific  date.^ 
Possible  targets  to  answer  the  question  of  comparison  include:  (1)  the 
agency’s  past  performance,  (2)  the  performance  of  similar  agencies  in 
similar  jurisdictions,  and  (3)  the  agency’s  own  performance  target.*^ 

Behn  also  addresses  the  value  and  difficulty  in  setting  performance  targets 
for  government  agencies  in,  “On  the  Executive  Responsibility  to  set 
Performance  Targets”.  He  illuminates  the  inherent  barriers  an 
organization  faces  in  developing  targets  and  making  them  public:  “Putting 
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a  number  and  a  date  in  the  same  sentence  is  a  commitment. .  the  more 
public  the  leaders  are  in  their  commitment  to  producing  X  by  Y,  the  more 
that  the  people  in  their  organization  will  understand  that  the  performance 
targets  are  serious.,  .yet  setting  such  targets  is  the  easiest  way  to  invite 
scrutiny  and  criticism.”^  Despite  these  impediments,  Behn  asserts  that, 
“citizens  need  to  establish  the  performance-target  ethic:  ‘We  expect  our 
elected  and  appointed  executives  to  establish  specific,  meaningful 
performance  targets.’  Then  citizens  will  have  a  basis  on  which  to  judge 
whether  their  government  is  performing  well.”^  Specific  and  accurate 
targets  are  essential  in  performance  measurement,  and  even  though 
publically  committing  to  them  can  be  challenging  for  city  officials, 
citizens  need  to  demand  that  the  targets  be  established  and  made 
transparent.  By  doing  this,  citizens  will  be  able  to  assess  government 
processes  in  a  more  accurate,  accountable,  transparent  and  accessible 
manner. 

The  need  for  emergency  preparedness  performance  measurement  at  the  local 
level 


While  the  U.S.  Department  of  Homeland  Security  tracks  performance  at  a 
national  level,  it  has  provided  little  guidance  to  state  and  local 
governments  on  how  to  systematically  and  appropriately  measure  the 
success  of  DHS-funded  initiatives  at  the  local  level.  Nonetheless,  DHS 
has  indicated  that  local  governments  should  be  tracking  such  progress,  and 
will  be  held  accountable  for  their  results.  To  make  matters  worse. 
Congress  often  directs  homeland  security  funds  toward  superfluous  or 
“pet”  projects,  limiting  local  progress  toward  achieving  minimum 
preparedness  goals.  This  section  details  the  rationale  for  developing  a 
local  performance  measurement  system,  as  well  as  a  method  to  help  local 
governments  begin  to  prioritize  critical  needs. 

Rationale  for  emergency  preparedness  performance  measurement  at  the 
local  level 

In  the  immediate  aftermath  of  the  tragedy  of  9/1 1,  Congress  and  the  White 
House  moved  quickly  to  direct  more  funding  toward  state  and  local 
homeland  security  efforts.  Fearing  that  local  governments  would  be 
resistant  to  specific  guidelines.  Congress  provided  much  of  the  funding 
through  block  grants  with  vague  requirements,  assuming  that  the  localities 
would  know  how  to  spend  the  money  to  reach  their  preparedness  goals. 

As  the  National  Preparedness  Goal  Homeland  Security  Presidential 
Directive  8  asserts. 


Senior  officials  at  each  level  of  government  are  ultimately 
accountable  to  their  citizens  for  assessing  preparedness  for 
their  respective  jurisdictions.  They  are  best  positioned  to 
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understated  that  assessment  data  may  not  be  all- 
encompassing  and  that  other  factors  weigh  into  the 
preparedness  equation.  They  will  retain  the  authority, 
responsibility,  and  discretion  to  provide  additional 
evidence  or  information  to  the  overall  assessment  for  their 
respective  entity.  'This  preserves  and  respects  the  flexibility 
of  Federal,  State,  local,  and  tribal  authorities,  who  are 
most  familiar  with  and  have  the  best  insight  into  their 
homeland  security  needs,  while  enstihng  accountability  for 
everyone's  role  in  national  preparedness.^ ' 

This  ambiguity  created  significant  difficulties  for  state  and  local 
governments  in  choosing  how  to  allocate  funds  and  tracking  the  impact  of 
their  efforts.  As  the  Century  Foundation  reported  in  The  Forgotten 
Homeland,  “While  billions  of  dollars  have  been  spent,  the  lack  of  explicit 
and  detailed  national  goals  for  local  governments  to  achieve  has  made  it 
impossible  to  know  how  close  we  have  come  to  the  minimum  defensive 
and  responsive  capabilities  we  need.”’^  MOEP  Director  Donald 
McGough  echoed  this  sentiment,  providing  one  example  about  emergency 
cots.  While  MOEP  purchased  a  certain  number  of  cots  for  the  Boston 
metropolitan  area  with  DHS  funds.  Director  McGough  lacks  the  tools 
needed  to  measure  the  impact,  benefit,  or  effectiveness  of  this  investment, 

*  •*13 

or  to  determine  whether  the  cots  translate  into  a  preparedness  capability. 

At  the  same  time,  he  is  held  accountable  by  the  Mayor’s  Office  in  Boston, 
the  State  of  Massachusetts,  DHS,  and  other  stakeholders  to  demonstrate 
effective  use  of  DHS  dollars  and  progress  toward  improved  security. 

Current  performance  measurement  efforts  in  the  City  of  Boston 

The  City  of  Boston  strives  to  provide  the  highest  level  of  services  to  its 
citizens.  Mayor  Menino’s  administration  is  dedicated  to  improving 
performance,  responsiveness  and  customer  satisfaction  across  all  city 
departments.  City  leaders  recognize  the  importance  of  timely,  reliable  and 
consistent  information,  and  thus  launched  the  performance  measurement 
system  Boston  About  Restdts  in  2006.  Integral  to  this  system  is  a  city¬ 
wide,  web-based  performance  measurement  application  which  aims  to 
closely  align  strategy  with  performance  measures,  improve  coordination 
of  citywide  initiatives,  and  develop  performance  scorecards  and  indicators 
of  success  by  department,  program  and  strategy. Boston  About  Results 
was  designed  to  inform  program  managers,  city  leaders  and  citizens 
regarding  the  levels  of  services  and  outputs  being  rendered  to  the  public. 

It  is  guided  by  the  following  performance  principles:  (1)  Improving 
Accountability  and  Transparency,  (2)  Focusing  on  the  Mission,  (3) 
Strengthening  Management,  and  (4)  Strategic  Resource  Planning.'^ 


8 


Rosenband 

Schroeder 


Measuring  Boston’s  Security  Investment 


One  specific  way  the  City  of  Boston  desires  to  increase  transparency  is  by 
placing  key  performance  results  on  the  website,  as  well  as  fostering 
communication  across  departments  about  the  levels  and  types  of  services 
that  are  being  administered  to  citizens.  The  measures  to  assess  the 
performance  of  city  programs  are  intended  to  tie  directly  to  the  overall 
mission  and  goals  of  the  city.  Status  indicators  were  developed  by  city 
leaders  to  with  the  aim  of  facilitating  managers  tracking  and  reviewing 
progress  against  stated  targets.  Below  is  an  example  of  status  indicators 
used  in  Boston  About  Results. 

Figure  I:  Status  indicators  for  managers 


Status  Indicators 

B  Pexfomuivc*  rneefe  or  exceeds  txiget 
^  Pexforavance  is  shg^htlv  belo'^v  target 
o  Meaiiire  is  treiidii\§  far  below 
needs  attenticu 

O  Measw:©  data  missing  or  no  target  created 


These  indicators  are  applied  to  programs  to  indicate  the  degree  to  which 
targets  are  being  met  and  identify  those  areas  that  require  remedial 
attention.  The  strategic  resources  planning  system,  to  which  Boston  About 
Results  is  an  integral  part,  provides  a  baseline  of  services  rendered  by 
department.  Below  is  the  Sustainability  and  Security  scorecard  portion  for 


FY08. 
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Figure  2:  Sustainability  and  Secttrity  Performance  Goals  FY08 


Sustainability  &  Security 
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Capital  and 
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%  of  roadway  miles 
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4% 

Street  trees  planted 

500 

Protect 

Homeowners 

Affordable  housing  units 
preserved  through  foreclosure 
prevention 

147 

Homebuyers  receiving 
foreclosure  prevention 
technical  assistance 

1532 

Finally,  budget  accountability,  where  financial  and  operational 
performance  is  monitored  against  the  established  budget  plan  and 
expected  service  levels,  as  well  as  the  Mayor’s  24  hour  Constituent 
Service  Hotline,  provide  measures  and  an  assessment  of  performance  that 
reflects  the  level  of  services  rendered  to  Bostonians,  which  in  turn 
measures  the  degree  to  which  the  city  is  meeting  its  mission. 

While  city  leadership  intended  for  Boston  About  Results  to 
comprehensively  measure  the  government’s  service  output,  the  system  has 
not  entirely  succeeded  yet.  Critics  address  the  performance  measurement 
system  and  contend  that  so  far,  it  has  not  met  its  stated  objectives.  As  the 
Boston  Herald  pointed  out. 

Other  data  also  raised  questions.  As  for  the  Fire 
Department,  the  mayor 's  Boston  About  Results  site  brags 
the  average  number  of  firefighters  absent  due  to  injury  is 
hovering  at  an  average  of  41  Jakes.  Payroll  records 
obtained  by  the  Herald  show  the  city  paid  out  $14. 7  million 
in  “injured’'  pay  last  year  to  some  630  firefighters. 

In  the  Police  Department,  the  city  /xiid out  $7.4  million  last 
year  in  injury  pay.  As  reported  in  the  Herald  last  week, 
police  paid  out  $47.8  million  last  year  in  overtime  -  a 
statistic  not  listed  on  the  mayor 's  About  Results  refxyrt. 
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Currently,  MOEP  performance  is  not  measured  or  included  in  Boston 
About  Results.  While  MOEP  may  ultimately  be  required  to  participate  in 
Boston  About  Results.,  the  agency  still  lacks  the  internal  mechanism  to 
measure  its  activities  and  the  effectiveness  of  DHS-funded  initiatives.  In 
an  interview  for  this  PAE,  MOEP  Regional  Planner  Atyia  Martin 
indicated  that  a  having  a  performance  measurement  system  such  as  the 
Balanced  Scorecard  would  help  MOEP  begin  to  contribute  monthly 

/  7 

performance  reports  to  Boston  About  Residts. 


Recommendation  1:  The  Balanced  Scorecard 


The  Balanced  Scorecard  successfully  met  each  of  the  evaluation  criteria 
used  in  this  PAE. 


SYSTEMS 

AND 

CRITERIA 

Simple 

format 

Objective 

measurement 

Forward- 

looking 

measurement 

More  than 

mere 

checklists 

Inexpensive 

Balanced 

Scorecard 

The  Balanced  Scorecard  method  is  a  holistic  approach  that  ties  the 
organization’s  objectives,  measures,  and  targets  to  their  overall  vision  and 
strategy.  Developed  in  the  1990s  by  Robert  Kaplan  and  David  Norton*^, 
the  Balanced  Scorecard  made  waves  in  the  private  sector,  replacing  simple 
financial  indicators  with  an  organization-wide  system  of  measurement. 
Today,  the  Balanced  Scorecard  is  used  by  close  to  half  of  Fortune  1000 
organizations  and  is  gaining  momentum  in  the  non-profit  and  public  sector 
arenas.’^ 

Figure  3  (below)  depicts  the  interrelation  between  measures,  objectives, 
strategy  and  the  vision  of  an  organization.  The  organization’s  vision 
determines  its  strategy,  objectives  and  performance  measures.  The  reverse 
relationship) — namely  measures  building  toward  a  vision — is  also  evident. 
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Figure  3:  Illustration  of  the  purpose  of  the  Balanced  Scorecartf^ 
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The  benefits  of  the  Balanced  Scorecard  are  numerous.  By  using  the 
Balanced  Scorecard,  leaders  are  able  to  communicate  their  overall  vision 
and  strategy  to  both  employees  and  external  stakeholders  who  might 
otherwise  have  only  partial  information.  This  is  of  particular  use  to  lower- 
level  employees  because  they  can  see  how  their  role  fits  into  the  “big 
picture”  strategy  of  the  organization.  Managers,  in  turn,  can  use  the 
Balanced  Scorecard  to  ensure  that  investments  are  tied  to  their  core 
mission,  and  that  employees  are  getting  the  training  they  need  to  move  the 
organization  forward.  Some  governments  find  it  helpful  to  link  their 
Balanced  Scorecard  directly  to  their  budget,  to  be  able  to  show  that  every 
expense  builds  toward  the  organization’s  ultimate  goals. 

In  Balanced  Scorecard:  Step-By-Step  for  Government  and  Nonprofit 
Agencies,  Paul  Niven  identifies  several  additional  benefits  of  the 
scorecard,  including  long-term  planning,  accountability  and  transparency, 
attracting  scarce  resources,  focusing  the  organization’s  mission,  driving 
change,  and  inspiring  trust. The  Balanced  Scorecard  serves  as  a 
measurement  system,  a  strategic  management  system,  and  a 
communication  tool. 


12 


Measuring  Boston’s  Security  Investment 


Rosenband 

Schroeder 


In  general,  Balanced  Scorecards  include  four  persf>ectives;  financial, 
customer,  learning  and  growth,  and  internal  processes.  In  the  private 
sector,  the  financial  perspective  receives  top  billing,  as  the  organization’s 
success  is  tied  to  its  bottom  line.  In  the  non-profit  and  public  sectors, 
however,  experts  recommend  that  the  Balanced  Scorecard  be  modified  to 
show  the  customer  perspective  as  the  leading  perspective.  Another  change 
that  must  occur  when  applying  the  Balanced  Scorecard  to  the  public  sector 
is  that  the  “Customer”  perspective  may  include  several  distinct  sets  of 
customers  -  including  citizens,  higher  governmental  authorities,  and  other 
stakeholders. 


In  developing  a  Balanced  Scorecard,  the  organization  must  first  derive  its 
mission,  vision,  and  values.  This  sets  the  stage  for  the  organization  to 
build  its  strategy.  Typically,  strategy  building  sessions  involve  a  SWOT 
assessment,  which  analyzes  the  organization’s  strengths,  weaknesses, 
opportunities  to  reach  their  strategic  goals,  and  threats  to  achieving  those 
outcomes.  This  helps  the  organization  define  their  strategic  issues  and 
construct  a  “Strategy  Map,”  which  is  defined  in  the  next  section. 

Once  the  organization  has  identified  its  strategy,  it  can  better  assign 
objectives  to  each  of  the  four  Balanced  Scorecard  perspectives.  For  the 
Balanced  Scorecard  to  serve  as  a  high-level  document,  it  is  best  to  have 
only  a  few  key  objectives  for  each  of  the  four  areas.  Once  objectives  have 
been  identified,  the  organization  can  then  assign  measures,  targets,  and 
new  initiatives  to  address  any  target  shortfalls.  In  developing  performance 
measures,  experts  recommend  a  mix  of  lead  indicators,  which  measure 
intermediate  processes  and  might  indicate  milestones  in  the  future,  and  lag 
indicators,  which  look  back  at  things  that  have  already  happened. In 
addition,  as  Niven  states,  “effective  metrics  provide  direction,  align 
employees,  improve  decision-making,  and  serve  as  a  basis  for  resource 
allocation  decisions.”^^ 


In  a  January  2009  meeting,  we  took  the  MOEP  Director  Donald  McGough 
and  MOEP  Regional  Planner  Atyia  Martin  through  the  exercise  of 
identifying  the  organization’s  vision,  strategy,  and  objectives  for  each 
balanced  scorecard  perspective.  From  this,  we  developed  a  Balanced 
Scorecard  for  the  organization,  shown  on  the  following  pages.  While  we 
were  able  to  identify  the  objectives  and  initial  performance  measures  for 
each  of  the  four  perspectives,  the  target  and  initiatives  information 
remains  classified,  and  will  be  filled  in  by  MOEP  at  a  later  date. 


Strategy  Map!^‘* 


A  strategy  map  is  a  visual  representation  of  how  an  organization  intends  to 
create  value  for  its  customers,  stakeholders,  employees,  and  citizens. 
Strategy  maps  represent  intricate  relationships  among  all  key  perspectives 
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that  are  used  in  the  Balanced  Scorecard — financial,  customer,  learning  and 
growth,  and  internal  processes.  It  is  vital  for  an  organization  to  understand 
the  connected  relationships  that  exist  between  objectives  and  operational 
measures  across  each  of  the  four  perspectives.  Cause  and  effect  are 
illustrated  via  arrows  to  represent  the  network  of  influences  and 
dependencies  to  achieving  one’s  strategy.  Strategy  maps  show 
employees  at  every  level  how  their  actions  contribute  toward  the 
organization  achieving  its  mission. 

Limitations  of  the  Balanced  Scorecard 

Niven  identified  “Eleven  Deadly  Sins  of  Balanced  Scorecard,”  which  fall 
into  three  categories:  people-related,  process-related,  and  technology- 
related.  In  general,  the  barriers  to  a  successful  Balanced  Scorecard  system 
result  from  poor  managerial  buy-in,  utilizing  quick  fixes  instead  of  long¬ 
term  strategy,  and  failing  to  properly  educate  employees  about  their  role  in 
the  scorecard  process.  The  tools  for  avoiding  some  of  these  pitfalls  can  be 
found  in  the  Balanced  Scorecard  Implementation  Plan  section  of  this  PAE. 

Figure  4:  Niven ’s  “Eleven  Deadly  Sins  of  Balanced  Scorecard 


1. 


Taking  the  time  to  gather  relevant  data  (a  technology  and  process 
challenge) 

Not  making  BSC  a  critical  part  of  management  process  (a  process 
challenge) 

Stopping  the  education  of  users  and  managers  (a  people  challenge) 
Looking  for  the  Aha!  instead  of  the  “I  knew  it!”  (a  process  challenge) 
Managing  understanding  and  support  (a  people  challenge) 

Not  fighting  the  freeloaders  who  resist  change  (a  people  challenge) 
Searching  for  push-button  solutions  (a  technological  challenge) 
Expecting  the  design  to  freeze  (a  technology  challenge) 

Assuming  no  hidden  costs  (a  resource  challenge) 

10.  Managing  from  the  executive  suite  (a  people  management  challenge) 

1 1 .  Forgetting  the  values,  vision,  and  mission  (a  process  challenge) 


2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 


Examples  of  an  MOEP  Strategy  Map  and  a  top-level  Balanced  Scorecard 
are  included  in  the  following  two  pages.  If  MOEP  decides  to  implement 
the  Balanced  Scorecard,  these  templates  can  be  used  to  begin  the 
scorecard  process. 
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The  Balanced  Scorecard  in  practice 


Utah  State  government 

The  State  of  Utah  has  long  been  a  leader  on  performance  measurement.  In 
fact,  in  2008,  Utah  moved  to  the  head  of  the  pack  and  gained  the  title  of 
best -managed  state  by  the  Pew  Center  on  the  States.  According  to 
Michael  Hansen,  Director  of  Strategy  and  Management  for  the  Utah 
Governor’s  Office  of  Planning  and  Budget,  this  title  was  given  in  part 
because  of  Utah’s  fiscal  conservativeness,  but  more  for  their  use  of  the 
Balanced  Scorecard,  which  links  strategy  to  budgeting. 

The  most  innovative  part  of  Utah’s  Balanced  Scorecard  approach  is  that 
the  Governor’s  Office  of  Planning  and  Budget  asks  all  employees  to  come 
up  with  performance  metrics  and  targets  for  themselves,  rather  than 
handing  these  metrics  down  from  the  top.  Hansen  argued  that  this  has 
significantly  increased  buy-in  to  the  Balanced  Scorecard  program,  and  has 
led  employees  to  set  greater  goals  for  themselves. 

[In  Utah  we  are]  trying  to  have  the  Balanced  Scorecard 
grow  organically,  and  we  want  individuals  to  develop  their 
own  measures.  People  are  setting  higher  goals  and  goals 
that  make  more  sense,  rather  than  arbitrary  goals.  All  the 
way  down  to  the  lowest  level  people  are  setting  their  own 
goals.  One  of  the  things  the  Balanced  Scorecard  does  is  it 
acts  as  a  mediating  entity  -  the  employee  can ’t  wait  for  the 
review.  The  real  big  thing  it  does,  the  endgame  of  the 
scorecard,  is  a  change  in  behavior. 

This  move  to  have  individual  employees  define  their  own  metrics, 
however,  was  not  Utah’s  original  plan.  In  fact,  Utah’s  Governor  initially 
directed  his  cabinet  members  to  implement  the  Balanced  Scorecard  top- 
down.  After  an  initial  burst  of  enthusiasm,  however,  most  employees  soon 
grew  to  resent  the  metrics  that  were  imposed  on  them.  As  time  went  on, 
officials  began  to  let  employees  develop  their  own  measures,  which  led  to 
far  greater  success. 

Another  innovation  within  Utah’s  Balanced  Scorecard  effort  is  to  allow 
agencies  to  come  up  with  their  own  scorecard  perspectives,  rather  than 
adhering  to  the  traditional  four  perspectives:  customer,  financial,  internal 
processes,  and  learning  and  growth.  As  Amy  Lightfoot,  Special  Programs 
Director  for  the  Utah  Department  of  Public  Safety  described,  this  can  help 
managers  and  employees  better  understand  the  scorecard  process,  and  how 
their  duties  can  translate  into  measures  and  goals. 

Initially,  I  tried  to  have  the  divisions  break  [the  Balanced 
Scorecard]  out  on  internal  bu.siness  processes,  and  the 
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categories  that  are  traditional  in  Balanced  Scorecard  but 
we  ran  into  a  lot  of  semantic  issues.  It  was  a  hang-up  for 
people  because  no  one  would  agree  what  an  external 
customer  was  or  an  internal  customer.  So  I  said  forget 
about  that  -  we  went  more  based  on  strategy.  What  I  found 
was  when  we  did  that,  we  still  had  the  measures  and  we 
could  still  rearrange  it  into  the  Balanced  Scorecard 
categories. 

Instead  of  using  the  traditional  Balanced  Scorecard  categories,  Utah 

Department  of  Public  Safety’s  Homeland  Security  Emergency  ; 

Management  Division  categorizes  its  Balanced  Scorecard  into  four 

sections:  Emergency  Preparedness,  Emergency  Response,  Emergency 

Recovery,  and  Emergency  Mitigation  (shown  below).  Their  experience 

might  help  inform  MOEP  planners  as  they  move  forward  in  adapting  a 

Balanced  Scorecard  system  for  the  Metro-Boston  area. 

I 

I 

I 

I 

( 
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At  present,  Utah  officials  collect  individual  Balanced  Scorecards  from  the 
state’s  24  agencies  each  month.  They  then  review  the  scorecards  and 
provide  feedback  to  the  agency  leadership.  As  Hansen  described,  his 
office  is  most  concerned  when  the  scorecards  come  back  to  them  with 
little  revision  each  month.  Their  goal  is  to  have  the  Balanced  Scorecard 
act  as  a  living  document,  with  employees  constantly  updating  metrics  and 
targets,  and  finding  ways  to  make  processes  more  efficient  in  their  own 
jobs. 


If  the  [agency  ’sj  Balanced  Scorecard  never  changes,  / 'm 
highly  suspect.  If  the  metrics  don  7  change,  I  know  they 
have  delegated  it  to  the  intern.  And  so  what  we  do  is  we 
have  a  monthly  thing  where  they  turn  it  in,  we  review  with 
senior  staff,  and  then  we  give  feedback. 

[I  see  the  Balanced  Scorecard  as J  facilitating  the  strategic 
conversation.  Sometimes  that  person  at  the  front  counter, 
they  have  an  idea  of  how  to  do  things  better.  The  Balanced 
Scorecard  helps  you  start  having  a  strategic  dialogue 
between  managers  and  worker s.^^ 

According  to  Hansen,  Utah’s  ultimate  goal  is  to  move  their  Balanced 
Scorecard  feedback  process  online-  much  like  the  private-sector 
“dashboard”  approach.  This  would  allow  the  Governor’s  staff  to  view 
agency  performance  on  important  metrics  every  day,  and  move  quickly  to 
correct  inefficiencies  in  process  and  spending. 

The  City  of  Charlotte 

The  City  of  Charlotte  is  widely  considered  the  first  success  story  of  a 
government  entity  implementing  the  Balanced  Scorecard  in  a  public  or  a 
nonprofit  setting.  Charlotte  put  the  scorecard  into  practice  in  1996  by 
contracting  with  a  consulting  firm  that  educated  and  trained  city  staff 
about  the  Balanced  Scorecard  and  how  to  develop  the  performance 
measurement  system.  City  leadership  then  ensured  employees  were 
educated  about  the  performance  measurement  system  by  discussing  the 
Balanced  Scorecard  in  the  organization’s  newsletter,  holding  training 
sessions,  publishing  a  glossary  of  terms  used  in  the  measurement  system, 
and  developing  a  Balanced  Scorecard  Handbook  which  contained  useful 
definitions  and  other  information. 

Before  the  Balanced  Scorecard,  900  measures  were  tracked  across  1 3  key 
business  units.  After  the  scorecard’s  integration,  the  number  of  measures 
dropped  to  260  across  all  key  units.  City  officials  who  were  involved  in 
the  scorecard’s  implementation  gave  the  following  advice  for  other 
governmental  organizations  striving  to  use  the  Balanced  Scorecard;  (1)  the 
scorecard  requires  significant  time  and  commitment  to  ensure  its 
successful  development  and  implementation,  (2)  one  cannot  over- 
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communicate  the  scorecard  goals  and  processes  to  employees,  (3)  a  high- 
level  champion  is  very  important  to  gaining  buy-in,  and  (4)  the  scorecard 
requires  an  iterative  process  that  continually  improves  its  effectiveness 
and  alignment  to  the  strategy.^'  Below  is  an  example  of  Charlotte’s 
corporate-level  scorecard  measures. 

Figure  5:  An  excerpt  from  the  City  of  Charlotte 's  Balanced  Scorecard^  ^ 


Perspective 

Objective 

Sample  Measure 

Target 

Serve  the  Customer 

Strengthen 

Neighborhoods 

Number  of  stable 
neighborhoods  as 
measured  by  the  Quality 
of  Life  index 

102  stable 
neighborhoods 

Run  the  Business 

Develop  Collaborative 
Solutions 

Percent  of  strategic 
transportation  and  land 
use  projects  utilizing 
integrated  land  use  and 
transportation  planning 

100% 

Manage  Resources 

Expand  Tax  Base  and 
Revenues 

Percent  change  in  tax 
valuation  in  targeted 
neighborhoods 

10%  increase  in  tax 
valuation 

Develop  Employees 

Recruit  and  Retain 
Skilled,  Diverse 
Workforce 

Percent  increase  in  city 
average  turnover  rate 

<5%  increase  in 
turnover 

The  City  of  Charlotte’s  successful  development  and  deployment  of  the 
Balanced  Scorecard  affirms  that  this  performance  measurement  system 
can  positively  impact  governmental  organizations  and  ensure  a  closer 
alignment  of  processes  and  budget  with  the  overall  strategy,  increasing 
efficiency  and  effectiveness  and  better  serving  key  stakeholders. 


Recommendation  2:  Focus  on  Clarke’s  critical  priorities 


While  homeland  security  dollars  are  flowing  to  state  and  local 
governments,  special  interest  and  pork  projects  have  limited  some 
localities  from  achieving  critical  minimum  preparedness  capabilities.  To 
remedy  this  situation,  national  security  expert  Richard  A.  Clarke  has 
proposed  a  Metropolitan  Area  Security  Plan  that  would  offer  cities  a 
standardized  and  prioritized  list  of  critical  homeland  security 
requirements.  Clarke’s  plan  provides  a  systematic  way  for  federal 
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government  to  work  with  state  and  local  governments  to  fund  critical 
needs  over  superfluous  and  wasteful  projects.  As  Clarke, 

This  concept  looks  at  each  metropolitan  area  holistically, 
determining  what  the  area  needs.  The  Federal  government, 
working  together  with  metropolitan  area  councils  and  the 
States,  could  then  decide  what  county,  city,  or  town  agency 
would  be  assigned  the  capability  on  behalf  of  the  entire 
region...  We  could  determine  how  much  it  would  cost  to 
carry  out  the  entire  effort  nationwide  in  three  years  or 
five...  We  could  decide  that  some  metropolitan  areas  would 
be  given  priority.  If  some  metropolitan  areas  had  unique 
requirements,  they  could  have  substituted programs.^^ 

While  Clarke’s  proposal  is  targeted  toward  federal  government  planners, 
the  theory  behind  his  argument  is  applicable  to  local  entities  such  as 
MOEP.  By  setting  priorities  through  a  performance  measurement  system 
such  as  the  Balanced  Scorecard,  MOEP  can  target  funds  toward  its  most 
critical  needs  and  track  their  progress  toward  meeting  those  goals.  This  is 
fundamentally  different  from  the  DHS’  Target  Capabilities  List  (TCL), 
which,  as  The  Forgotten  Homeland  states,  “only  suggests  that  a  capability 
should  exist,  rather  than  designating  what  level  must  be  achieved,  by 
when,  and  in  what  order.  Using  the  TCL,  a  jurisdiction  could,  for 
example,  decide  to  buy  tow  trucks  with  Federal  block  grants  without  first 
having  achieved  interoperable  radio  communications  for  emergency 
responders. 

Using  Richard  A.  Clarke’s  standardized  priorities  list,  local  entities  can 
move  beyond  the  Target  Capabilities  List  and  pork  projects  toward  a 
transparent  and  fiscally  sound  system  of  meeting  basic  critical  needs. 
Below  is  the  list  of  the  minimum  requirements  for  Clarke’s  Metropiolitan 
Area  Security  Plan.  This  can  assist  MOEP  planners  as  they  assess  what 
Boston  has  already  accomplished,  and  set  future  priorities  through  a 
performance  measurement  system. 
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Figure  6:  Clarke  \s  Metropolitan  Area  Security  Plan:  Sorted  by  order  of  timeframe^^ 

Within  One  Year 

•  Conduct  security  awareness  campaigns  and  upgrade  public  address  systems  on  public 
transportation  systems 

•  Initiate  random  security  checks  on  rail  and  subway  systems 

•  Aehieve  a  standard  level  of  uniformed,  K-9,  and  plainclothes  police  officers  at  rail  and 
subway  stations 

•  Create  and  train  a  local  police  counterterrorism  unit  with  community  outreach  and  civilian 
oversight,  added  needed  officers  to  the  force 

•  Develop  local  law  enforcement  intelligence  capabilities  to  bring  community  policing 
activities  and  national  counterterrorism  efforts  together 

•  Create  and  test  local  government  crisis  management  and  incident  response  systems  to  a 
national  standard  within  one  year 

Within  Two  Years 

•  Establish  an  operational  evacuation  system  for  cities  and  neighborhoods 

•  Create  a  secure  communications  system 

•  Achieve  a  standard  level  of  heavy  rescue  capability 

•  Establish  intelligent  video  security  systems  for  key  neighborhoods  and  infrastructures,  with 
appropriate  civil  liberties  and  privacy  safeguards 

•  Establish  effective  public  communications  systems  (including  public  address,  reverse  911, 
email) 

•  Conduct  training  exercises  to  a  national  standard 

Within  Three  Years 

•  Install  intelligent  video  security  systems  at  all  rail  and  subway  stations 

•  Upgrade  communications  and  rescue  capability  for  rail  and  subway  systems,  including 
tunnels,  to  a  standard  level 

•  Create  a  reliable,  interoperable  metropolitan  communications  system,  including  State  and 
Federal  agencies 

•  Achieve  an  agrccd-upon  level  of  training  and  equipping  for  first  responder  capability  to  deal 
with  chemical,  biological,  and  radiological  incidents 

•  Achieve  a  surge  ability  to  perform  a  given  level  of  emergency  medical  treatment,  including 
for  mass  trauma,  bums,  and  WMD-rclated  incidents 

•  For  a  given  ration  to  the  population,  develop  the  capability  to  add  and  staff  additional  patient 
beds  in  the  event  of  a  mass-casualty  or  pandemic  event 

•  Identify  and  mitigate  vulnerabilities  at  chemical  plants,  chemical  railcar  sidings,  radiological 
material  facilities,  tunnels,  bridges,  arenas,  and  stadiums  to  national  levels 


In  addition  to  utilizing  this  list,  cities  can  also  build  a  list  of  priorities 
based  on  a  threat  analysis.  R.P.  Eddy,  CEO  of  Ergo  and  Senior  Fellow  for 
Counterterrorism  at  the  Manhattan  Institute,  described  a  high-level  process 
that  cities  can  use  to  determine  their  critical  priorities. His  five-step 
process  is  detailed  below. 
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1 .  Figure  out  what  the  threats  are  to  the  city 

2.  Determine  the  most  likely  scenarios  for  attack 

3.  Determine  how  your  city  might  be  used  as  a  proxy  for  a  threat  to 
another  city,  i.e.  as  a  staging  site  for  terrorists 

4.  Determine  the  things  that  are  necessary  to  prepare,  prevent,  and 
respond  to  attacks 

5.  Build  your  preparation  against  terrorist  threats  in  a  way  that  helps 
prepare  for  other  situations  (all-hazards  approach):  riots,  avian  flu, 
mudslides,  earthquakes,  etc. 

In  Eddy’s  expert  opinion,  community  policing  should  be  high  on  the  list  of 
city  priorities.  As  Eddy  said: 

Try  to  educate  your  community  increase  the  eyes  on  the 
target  through  ad  campaigns,  threat  camfxiigns,  and 
community  defense,  such  as  “if  you  see  something,  say 
something.  ”  [This  is  the  best]  in  terms  of  hang  for  the 
huck.^^ 

Eddy  further  expanded  upon  resource  allocation,  noting  that  the  majority 
of  resources  should  be  spent  on  point  four  of  the  five-step  process  above 

This  is  where  the  lion's  share  of  resources  must  he  spent.  A 
thorough  implementation  and  review  process  will  be 
necessary  to  answer  each  of  these  three  needs  (prepare, 
prevent,  respond)  across  all  likely  scenarios  and  generic 
threats.^^ 

Focusing  on  addressing  critical  security  priorities  is  the  only  way  to  ensure 
a  city  is  able  to  prevent,  prepare  for,  and  respond  to  terrorist  attacks  or  all¬ 
hazard  threats.  Inherent  in  determining  those  priorities  is  an  effective 
assessment  of  risk,  as  described  by  Eddy  above.  Working  in  tandem, 
these  techniques  provide  a  strong  basis  for  cities  to  begin  identifying  and 
addressing  critical  preparedness  priorities. 


Recommendation  3:  Incorporate  quality  DHS  measures 

Though  DHS  does  not  employ  the  Balanced  Scorecard,  the  performance 
measures  that  it  uses  tend  to  fall  into  one  of  the  four  typical  Balanced 
Scorecard  perspectives.  Many  of  the  measures  DHS  has  chosen  have  been 
developed  through  iterative  processes,  and  we  believe  that  these  are 
meaningful  indicators  that  can  be  incorporated  into  the  Balanced 
Scorecard.  Based  on  our  review  of  the  Fiscal  Year  2007-2009  DHS 
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Annual  Performance  Report,  we  were  able  to  identify  the  basic  types  of 
performance  measures  that  fell  under  each  perspective. 

Overall  Perspective 

1 .  Number  of  threats  in  a  certain  timeframe 

2.  Comparisons  to  nationwide  preparedness  percentages 

Customer  Perspective 

1 .  Percentage  of  secured  area 

2.  Percentage  of  apprehensions/screenings  at  designated  checkpoints 

3.  Percentage  of  legal  cases  that  result  in  conviction 

4.  Removal  rate  for  illegal  materials 

5.  Number  of  spot  checks  performed 

6.  Percentage  of  targeted  stakeholders  that  obtain  needed  security 
services 

7.  Percentage  of  the  population  covered  by  certain  security 
safeguards 

8.  Percentage  of  critical  infrastructure  secured 

9.  Five-year  average  of  deaths  and  injuries 

10.  Average  time  it  takes  to  provide  essential  logistical  services  to  an 
impacted  community 

1 1 .  Percentage  satisfaction  of  partner  organizations 
Financial  Perspective 

1.  Financial  loss  prevented  in  dollars 
Internal  Processes  Perspective 

1 .  Percentage  of  workers  performing  essential  duties 

2.  Percentage  of  information  available  to  those  who  need  it 

3.  Percentage  compliance  with  State  and  Federal  regulations 

4.  Time  it  takes  to  process  key  forms 

5.  Emergency  call  completion  rate 

6.  Number  of  planning  scenarios  completed 

7.  Number  of  internal  control  processes  tested  for  design  and 
operational  effectiveness 

Learning  and  Growth  Perspective 


1. 

2. 

3. 


Number  of  persons  trained  for  certain  emergency  situations 
Percentage  of  employees  performing  above  a  certain  level 
Percentage  of  capabilities  performed  acceptably  in  exercises 
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4.  Percentage  of  responders  who  say  they  are  better  prepared  because 
of  training 


Recommendation  4:  Undergo  accreditation  within  3  years 


Emergency  Management  Accreditation  Program  (EMAP) 

In  addition  to  the  in-house  techniques  such  as  the  Balanced  Scorecard, 
there  are  also  national  emergency  management  accreditation  programs  that 
provide  guidance  on  measuring  the  performance  of  security  initiatives. 

U.S  Representative  Heather  Wilson  from  New  Mexico  raised  the  topic  of 
accreditation  programs  in  an  interview  for  this  PAE.  She  suggested  a 
review  of  accreditation  programs  as  a  means  to  measure  the  security 
investment  Boston  has  made.^^  The  Emergency  Management 
Accreditation  Program  is  a  “voluntary  accreditation  process  for  State, 
regional,  territorial,  tribal,  county  and  municipal  government  programs 
that  coordinate  and/or  provide  activities  related  to  disaster  prevention, 
mitigation,  preparedness,  response,  and  recovery.  EMAP,  as  an 
independent  public  nonprofit  organization,  fosters  excellence  and 
accountability  in  emergency  management  and  homeland  security 
programs  by  establishing  credible  standards  applied  in  a  peer  review 
accreditation  process.”** 

The  Emergency  Management  Standard  is  a  tool  designed  for  iterated  use 
by  States  as  part  of  EMAP’s  voluntary  accreditation  processes.  The  steps 
to  accreditation  are  as  follows:  (1)  subscription  at  a  rate  of  $450/year,  (2) 
self-assessment  via  the  online  EMAP  Program  Assessment  Tool,  (3) 
submit  the  application  with  a  fee  of  $4,500  for  the  city  of  Boston,  (4)  on¬ 
site  assessment  by  an  EMAP  Commission  consisting  of  10  impartial 
officials,  (5)  committee  review,  and  (6)  accreditation  decision.  In 
addition,  the  costs  associated  with  the  on-site  assessment  must  be  covered 
by  the  local  emergency  management  program.**' 

The  EMAP  Program  provides  benchmarks  for  program  management  and 
operations  and  validates  professional  capabilities.  EMAP  considers  the 
following  program  elements  to  be  vital  components  to  any  program,  and 
thereby  focuses  accreditation  and  assessment  efforts  in  the  following 
areas: 
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Figure  7;  FMAP  vital  program  elements*^ 
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In  2003,  the  EMAP  Commission  agreed  to  work  with  the  DHS  Emergency 
Preparedness  and  Response/  Federal  Emergency  Management  Agency 
(EPR/FEMA)  to  conduct  baseline  assessments  of  states  and  territories. 
Currently,  the  second  round  of  assessments  is  underway  to  measure 
progress  states  have  made.'*^ 

The  Boston  Mayor’s  Office  of  Emergency  Preparedness  has  researched 
the  EMAP  program  thoroughly  and  intends  to  pursue  this  accreditation 
method  in  the  future.  Currently,  there  are  barriers  that  make  such  an 
endeavor  infeasible  for  the  organization.  For  example,  MOEP  lacks  a 
recovery  plan  and  will  need  to  devise  one  before  the  accreditation  process 
begins.  Furthermore,  the  financial  cost  associated  with  the  EMAP 
program  presents  difficulties  for  MOEP  due  to  funding  constraints.  The 
office  receives  roughly  95%  of  its  funding  from  the  Department  of 
Homeland  Security,  and  regulations  prohibit  that  money  from  being  used 
for  accreditation  purposes.  Nonetheless,  MOEP  is  committed  to 
undergoing  the  EMAP  process  in  the  future,  and  we  would  recommend 
this  endeavor. 

In  discussion  with  the  Utah  Department  of  Public  Safety,  Special 
Programs  Director  Amy  Lightfoot  indicated  that  the  State  of  Utah  was  the 
eighth  state  to  undergo  EMAP  accreditation,  and  that  they  continue  to 
report  their  status  as  a  performance  measure  to  the  Utah  Governor’s 
Office. 
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Other  Systems  Researched  and  Considered 

At  the  beginning  of  this  project,  MOEP  requested  information  regarding 
different  methods  and  tools  presently  being  used  by  other  government 
agencies  to  assess  performance  measurement.  This  section  describes  other 
measurement  systems  that  are  currently  in  use,  but  do  not  fit  the  criteria 
we  used  in  determining  the  policy  recommendations  of  this  PAE.  In 
general,  the  following  systems  fail  in  meeting  one  or  more  of  the 
following  criteria;  simple  format,  objective  measurement,  forward-looking 
measurement,  and  more  than  mere  checklists. 

U.S.  Department  of  Homeland  Security 

The  performance  measurement  system  used  by  the  Department  of 
Homeland  Security  fails  to  succeed  in  two  criteria:  simple  format  and 
forward-looking  measures. 


SYSTEM 

AND 

CRITERU 

Simple 

format 

Objective 

measurement 

Forward- 

looking 

measurement 

More  than 

mere 

checklists 

Inexpensive 

DHS  System 

As  America’s  newest  agency,  the  U.S.  Department  of  Homeland  Security 
(DHS)  understands  the  importance  of  measuring  performance.  Created 
by  the  Homeland  Security  Act  of  2002,  DHS  absorbed  the  functions  of  22 
Federal  agencies,  each  with  their  own  mission  and  agenda.  In  addition, 
DHS  is  beholden  to  more  than  80  congressional  committees  and 
subcommittees.  With  this  need  to  provide  broad  accountability,  DHS 
recently  began  the  process  of  developing  appropriate  agency-level 
performance  measures. 

The  DHS  system  of  performance  measurement 

The  DHS  system  of  performance  measurement  seeks  to  tie  overall 
department-level  goals  to  specific  programs  within  the  22  former  agencies 
Specifically,  the  five  main  DHS  goals  are  (1)  Protect  our  Nation  from 
Dangerous  People,  (2)  Protect  our  Nation  from  Dangerous  Goods,  (3) 
Protect  Critical  Infrastructure,  (4)  Build  a  Nimble,  Effective  Emergency 
Response  System  and  a  Culture  of  Preparedness,  and  (5)  Strengthen  and 
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Unify  DHS  Operations  and  Management."*^  Program  managers  are 
responsible  for  providing  quarterly  updates  to  their  performance  measures. 

The  Department  of  Homeland  Security  provides  several  layers  of  analysis 
for  each  performance  goal.  First,  DHS  provides  a  high-level  view  by 
rating  each  of  its  programs  with  either  a  blue  circle,  indicating  the 
program  has  met  75%  or  more  of  its  performance  targets,  a  green  triangle 
if  50-75%  of  performance  targets  are  met,  and  an  orange  square  if  the 
program  has  achieved  less  than  50%  of  its  performance  targets.  Figure  8 
(below)  details  this  measurement  system. 

Figure  8:  DHS  Performance  Goals^^ 

Table  16.  Goal  4.  Objective  4.1;  Succe$$  iu  Achie^ng  Performance  Goals 


y  Progniai  Performance  Goal 

Rating 

wms 

Bitot 

Gi  auts  (FEMA):  Enhance  the  Nation’s  prqjaredness  by  increasing  the  capability  of 

States,  temtories,  and  local  jurisdictions  to  prevent,  protect,  respond,  and  recover  from 
terrorism  and  all-hazard  incidents 

• 

S3.S253 

^fitigarion  (FEMA):  Reduce  the  impact  of  natural  hazards  on  people  and  property 
tlirough  the  analysis  and  reduction  of  risks  and  the  provision  of  flood  insurance. 

• 

S4.19S  1 

National  Preparedness  (FE^L4):  Improve  the  Nation's  ability  to  prepare  for.  respond  to, 
and  recover  from  acts  of  terrorism,  natural  disasters,  or  other  emergencies  through  exercise 
facilitation  implementation  of  the  National  Incident  Management  System,  and  the 
provision  of  emergency  management  training 

• 

S4S7  I 

U.S,  Fire  Administration  (FEMA):  Reduce  the  effect  of  fire  and  all  hazard  emergencies 
by  supporting  and  enhancing  the  delivery*  of  State  and  local  fire  and  emergency  services 
and  promoting  public  awareness. 

• 

S77.: 

Law  Enfoixeinent  Training  (FLETC):  Provide  law  enforcement  agents  and  officers, 
skilled  m  the  latest  techniques,  to  enforce  laws  and  regulations,  protect  theNatioa  and 
interact  with  the  public  w^xh  respect  for  indhiduals  and  civil  llb^^ 

A 

S28S7 

Medical  and  Biodefeuse  Programs  (OEL\):  Bolster  the  Nation's  biodefense  readiness 
by  enhancmg  the  national  architecture  to  rapidly  detect,  characterize,  and  respond 
effectively  to  a  large-scale  biological  event. 

S23  3 

Cotninntid.  Conn  ol  .ind  Interoperability  (.S&T);  Improve  and  develop  operable  and 
mteroperable  communications  for  emergency  responds;  develop  tools  to  improve  the 
secuntv’  and  integrity  of  the  mtemet,  and  improve  and  develop  automated  capabilities  to 
recogmze  potential  threats  through  science  and  technologv’ 

• 

S2S7 

Note  Blue  (•)  is  adaesied  b,-  meeting  75*.  oi  more  of  perforaance  targets  green  (A)  50  to  "4^^  and  wange  (M)  l«s  than  50°* 


The  second  layer  of  analysis  is  at  the  program  level.  Here,  DHS  details 
the  specific  measures  for  each  program  performance  goal,  indicating  the 
measure  and  whether  or  not  the  agency  has  achieved  that  measure  over  the 
last  four  years.  If  the  program  has  not  met  the  specified  performance 
target,  DHS  indicates  the  corrective  action  that  it  is  taking  to  meet  that 
target  in  the  future.  The  figure  below  is  useful  because  it  depicts  the  target 
and  results  for  a  specific  program,  indicating  that  a  program  met  its  target 
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in  FY04  and  FY05,  but  not  in  FY06  and  FY07.  In  FY08,  this  particular 
performance  measure  was  retired  and  will  likely  be  replaced  by  an 
alternative  measure. 

Figure  9:  l\xample  of  a  DHS  performance  measnremetU  tahle^^ 
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While  DHS’  measurement  system  is  thorough,  its  format  is  cumbersome 
and  it  provides  few  forward-looking  measures  to  track  intermediate  steps 
toward  longer-range  goals. 

Specific  DHS  performance  measures  for  state  and  local  government 

The  majority  of  DHS-level  performance  measures  that  refer  to  state  and 
local  government  fall  under  Goal  4,  “Build  a  Nimble,  Effective 
Emergency  Response  System  and  a  Culture  of  Preparedness.” 

The  following  is  a  list  of  state  and  local-centered  performance  measures 
under  the  Federal  Emergency  Management  Agency  (FEMA)  Grants 
program. 

1 .  Percent  of  analyzed  capabilities  performed  acceptably  in 
preparedness  and  response  exercises 

2.  Percent  of  grantees  reporting  significant  progress  toward  the  goals 
and  objectives  identified  in  their  state  homeland  security  strategies 

3.  Percent  of  urban  area  grant  recipients  reporting  significant  progress 
toward  identified  goals  and  objectives 

4.  Percent  of  significant  progress  toward  implementation  of  National 
Preparedness  Priorities 

5.  Percent  of  states  and  territories  accredited  by  the  Emergency 
Management  Accreditation  Program 

6  Percent  reduction  in  firefighter  injuries  in  jurisdictions  receiving 
Assistance  to  Firefighter  Grants  funding  compared  to  the  national 
average 

It  is  important  for  MOEP  to  be  aware  of  the  performance  goals  DHS  is 
using  at  the  federal  level,  as  they  can  help  inform  the  measures  MOEP 
chooses  to  implement. 

DHS  Lessons  Learned  Information  Sharing  website 
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The  Department  of  Homeland  Security  saw  a  critical  need  to  consolidate 
the  lessons  learned  by  homeland  security  professionals  across  the  nation 
and  make  them  readily  accessible  to  officials  working  the  same  issues  in 
different  areas.  The  valuable  best  practices  and  lessons  learned  had 
become  buried  in  the  cumbersome  official  after-action  reports  and  were 
not  leveraged  to  improve  operations  and  learning  across  jurisdictions  and 
cities.  In  response,  the  FEMA  Office  of  Grants  &  Training  under  DHS 
directed  the  National  Memorial  Institute  for  the  Prevention  of  Terrorism  in 
Oklahoma  City,  OK  to  develop  a  website  that  would  consolidate  and 
integrate  lessons  learned,  innovative  ideas  and  best  practices  for  the 
emergency  response  community.  In  addition  to  providing  these 
capabilities,  the  Lessons  learned  Information  Sharing  (LLIS.gov)  website 
is  the  national  repository  for  hundreds  after-action  reports,  as  well  as  a 
vast  library  of  templates  that  officials  can  use  to  produce  plans, 
procedures,  templates  and  tools.  LLIS.gov  uses  strong  security  and 
encryption  measures  to  protect  all  information  on  the  network.'*^  The 
website  has  facilitated  the  communication  of  lessons  learned  and  valuable 
experiences  that  aim  to  improve  homeland  security  activities  across  the 
nation.  While  this  is  an  effective  medium  to  communicate  best  practices 
regarding  preparedness  and  response  activities,  little  to  no  little 
information  is  shared  on  performance  measurement  practices  for 
homeland  security  and  emergency  preparedness  functions.  In  an  interview 
for  this  PAE,  Atyia  Martin  described  the  limitations  of  the  LLIS.gov 
website. 

[LLIS.gov]  has  been  helpful  in  terms  of  higher-level  federal 
guidance,  and  there  is  some  information  about  best 
practices  -  but  not  in  terms  of  metrics.  [It  also  does  not] 
represent  the  entire  universe  of  what  people  are  doing. 

Even  if  it  was  updated  regidarly,  there  would  be  some 
challenges  in  terms  of  haw  much  people  are  willing  to 
share. ^ 

In  sum,  while  LLIS.gov  provides  certain  useful  information  regarding 
emergency  preparedness,  it  has  contributed  little  in  terms  of  helping  cities 
define  performance  measures  or  measurement  systems. 

Office  of  Management  and  Budget 

The  measurement  system  used  by  the  Office  of  Management  and  Budget 
lacks  objective  measurement,  forward-looking  measures,  and  often 
resembles  mere  checklists. 
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